2024-09-16T03:45:20.280 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T03:45:20.302 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:45:20.385 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926 branch: squid description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905926' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9KCYuOYbk0CeaPowp25kaJj7YHOWaP7IJyUKpC+R3+hb0o1glDKlwrK/VGf6oKS1fGQV4rFhCtUbyo4PKoba8= smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFtJOv5Tu6JY3oQEQHbuJEnyvd1pZauNX4nuRRytC+gBT/8E2EKqQ/oU6CjndNj+VcUxIMLxUa0ANqOF7S7R6s= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3wo++gdDXP/6ydD3cgPDb79KlgLK6IKEdLgVMSNrMzZqk6S0fhjq/L6XcSBIn4MeGnJkj/eomOa5r0/7N7YIk= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOPrh9WaFxQsMRthRp+49kzCOBVBzhFmlvlc1a/UdPRKhO/ZldCnyL6g14sw28Rh6Zhi/8dR4VPUtvJ7HObjAQg= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T03:45:20.385 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T03:45:20.386 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T03:45:20.386 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T03:45:20.387 INFO:teuthology.task.internal:Checking packages... 2024-09-16T03:45:20.409 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T03:45:20.409 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T03:45:20.409 INFO:teuthology.packaging:ref: None 2024-09-16T03:45:20.410 INFO:teuthology.packaging:tag: None 2024-09-16T03:45:20.410 INFO:teuthology.packaging:branch: squid 2024-09-16T03:45:20.410 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:45:20.410 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-16T03:45:20.604 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:45:20.607 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T03:45:20.613 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T03:45:20.614 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T03:45:20.630 INFO:teuthology.task.internal:Saving configuration 2024-09-16T03:45:20.648 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T03:45:20.654 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T03:45:20.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:38:47.411436', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDQDoL/BNGU/EBDKxOEORQCj0B++/eVU/y6thhu/Y9qP'} 2024-09-16T03:45:20.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:38:47.414083', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFrr74HPGhpVeGAsdFM3soOnYm4M82bjOfaj0HVMjU7T'} 2024-09-16T03:45:20.725 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:38:47.412831', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILtdxo5Vp9BBclbtyVm9+HeQc//GADsqvvaNyQBy7+48'} 2024-09-16T03:45:20.749 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:38:47.409988', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOmIDlxv0yc45keywzDahYRo771IjCTVPUwwnBl51NIS'} 2024-09-16T03:45:20.749 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T03:45:20.755 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-16T03:45:20.755 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-16T03:45:20.756 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-16T03:45:20.756 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-16T03:45:20.756 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T03:45:20.858 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f639b962e60>, signals=[15]) 2024-09-16T03:45:20.858 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T03:45:20.864 INFO:teuthology.task.internal:Opening connections... 2024-09-16T03:45:20.864 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T03:45:20.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:45:20.947 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T03:45:20.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:45:21.030 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T03:45:21.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:45:21.111 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T03:45:21.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:45:21.189 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T03:45:21.195 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-09-16T03:45:21.201 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-09-16T03:45:21.201 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-09-16T03:45:21.247 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T03:45:21.247 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="22.04" 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=jammy 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T03:45:21.248 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=jammy 2024-09-16T03:45:21.249 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-09-16T03:45:21.276 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-09-16T03:45:21.282 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-09-16T03:45:21.283 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-09-16T03:45:21.328 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T03:45:21.328 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Ubuntu" 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="22.04" 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_CODENAME=jammy 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:ID=ubuntu 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE=debian 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T03:45:21.329 INFO:teuthology.orchestra.run.smithi039.stdout:UBUNTU_CODENAME=jammy 2024-09-16T03:45:21.330 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-09-16T03:45:21.351 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-09-16T03:45:21.356 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-09-16T03:45:21.357 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-09-16T03:45:21.403 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T03:45:21.403 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="Ubuntu" 2024-09-16T03:45:21.403 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="22.04" 2024-09-16T03:45:21.403 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T03:45:21.403 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_CODENAME=jammy 2024-09-16T03:45:21.403 INFO:teuthology.orchestra.run.smithi117.stdout:ID=ubuntu 2024-09-16T03:45:21.403 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE=debian 2024-09-16T03:45:21.403 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T03:45:21.404 INFO:teuthology.orchestra.run.smithi117.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T03:45:21.404 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T03:45:21.404 INFO:teuthology.orchestra.run.smithi117.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T03:45:21.404 INFO:teuthology.orchestra.run.smithi117.stdout:UBUNTU_CODENAME=jammy 2024-09-16T03:45:21.404 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-09-16T03:45:21.426 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-09-16T03:45:21.432 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-09-16T03:45:21.432 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-09-16T03:45:21.478 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T03:45:21.478 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="Ubuntu" 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="22.04" 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_CODENAME=jammy 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:ID=ubuntu 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE=debian 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T03:45:21.479 INFO:teuthology.orchestra.run.smithi188.stdout:UBUNTU_CODENAME=jammy 2024-09-16T03:45:21.480 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-09-16T03:45:21.509 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T03:45:21.519 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T03:45:21.525 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T03:45:21.525 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:45:21.528 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:45:21.530 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:45:21.533 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:45:21.537 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T03:45:21.542 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T03:45:21.542 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:45:21.577 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:45:21.579 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:45:21.581 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:45:21.594 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T03:45:21.695 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T03:45:21.709 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T03:45:21.709 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T03:45:21.710 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:45:21.711 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:45:21.712 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:45:21.712 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:45:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:45:21.717 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-09-16T03:45:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:45:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:45:21.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:45:21.721 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-09-16T03:45:21.722 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-16T03:45:21.722 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-16T03:45:21.767 INFO:teuthology.orchestra.run.smithi117.stdout:5.15.0-56-generic 2024-09-16T03:45:21.767 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.15.0-56-generic 2024-09-16T03:45:21.767 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get clean 2024-09-16T03:45:21.769 INFO:teuthology.orchestra.run.smithi037.stdout:5.15.0-56-generic 2024-09-16T03:45:21.770 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.15.0-56-generic 2024-09-16T03:45:21.770 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi188.stdout:5.15.0-56-generic 2024-09-16T03:45:21.772 INFO:teuthology.task.kernel:Running kernel on smithi188: 5.15.0-56-generic 2024-09-16T03:45:21.772 DEBUG:teuthology.orchestra.run.smithi188:> sudo apt-get clean 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-56-generic 2024-09-16T03:45:21.773 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.15.0-56-generic 2024-09-16T03:45:21.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-16T03:45:21.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2024-09-16T03:45:21.897 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-09-16T03:45:21.901 DEBUG:teuthology.orchestra.run.smithi188:> sudo apt-get update 2024-09-16T03:45:21.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-16T03:45:22.036 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:45:22.038 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T03:45:22.039 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T03:45:22.116 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi188.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T03:45:22.232 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T03:45:22.236 INFO:teuthology.orchestra.run.smithi188.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T03:45:22.238 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T03:45:22.290 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T03:45:22.391 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T03:45:22.395 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T03:45:22.398 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T03:45:22.398 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T03:45:22.420 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T03:45:22.420 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T03:45:22.420 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T03:45:22.427 INFO:teuthology.orchestra.run.smithi117.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T03:45:22.434 INFO:teuthology.orchestra.run.smithi117.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T03:45:22.440 INFO:teuthology.orchestra.run.smithi117.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T03:45:22.445 INFO:teuthology.orchestra.run.smithi117.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T03:45:22.446 INFO:teuthology.orchestra.run.smithi117.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T03:45:22.466 INFO:teuthology.orchestra.run.smithi117.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T03:45:22.466 INFO:teuthology.orchestra.run.smithi117.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T03:45:22.466 INFO:teuthology.orchestra.run.smithi117.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T03:45:22.466 INFO:teuthology.orchestra.run.smithi117.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T03:45:22.469 INFO:teuthology.orchestra.run.smithi117.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T03:45:22.469 INFO:teuthology.orchestra.run.smithi117.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T03:45:22.470 INFO:teuthology.orchestra.run.smithi117.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T03:45:22.471 INFO:teuthology.orchestra.run.smithi117.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T03:45:22.496 INFO:teuthology.orchestra.run.smithi117.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi117.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi117.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T03:45:22.507 INFO:teuthology.orchestra.run.smithi117.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi117.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi117.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi117.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi117.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi117.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi117.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi117.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi117.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi117.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T03:45:22.544 INFO:teuthology.orchestra.run.smithi117.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi117.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T03:45:22.546 INFO:teuthology.orchestra.run.smithi117.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T03:45:22.548 INFO:teuthology.orchestra.run.smithi117.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T03:45:22.549 INFO:teuthology.orchestra.run.smithi117.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T03:45:22.587 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T03:45:22.592 INFO:teuthology.orchestra.run.smithi188.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T03:45:22.593 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T03:45:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T03:45:22.614 INFO:teuthology.orchestra.run.smithi188.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T03:45:22.619 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T03:45:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T03:45:22.730 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T03:45:22.731 INFO:teuthology.orchestra.run.smithi188.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T03:45:22.904 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T03:45:22.936 INFO:teuthology.orchestra.run.smithi188.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T03:45:22.936 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T03:45:23.036 INFO:teuthology.orchestra.run.smithi188.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T03:45:23.040 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T03:45:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T03:45:23.052 INFO:teuthology.orchestra.run.smithi188.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T03:45:23.056 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T03:45:23.061 INFO:teuthology.orchestra.run.smithi188.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T03:45:23.061 INFO:teuthology.orchestra.run.smithi188.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T03:45:23.063 INFO:teuthology.orchestra.run.smithi188.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T03:45:23.065 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T03:45:23.065 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T03:45:23.067 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T03:45:23.077 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T03:45:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T03:45:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T03:45:23.086 INFO:teuthology.orchestra.run.smithi188.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T03:45:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T03:45:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T03:45:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T03:45:23.094 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T03:45:23.099 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T03:45:23.099 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T03:45:23.100 INFO:teuthology.orchestra.run.smithi188.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T03:45:23.101 INFO:teuthology.orchestra.run.smithi188.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T03:45:23.159 INFO:teuthology.orchestra.run.smithi188.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T03:45:23.163 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T03:45:23.163 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T03:45:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T03:45:23.169 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T03:45:23.170 INFO:teuthology.orchestra.run.smithi188.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T03:45:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T03:45:23.176 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T03:45:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T03:45:23.200 INFO:teuthology.orchestra.run.smithi188.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T03:45:23.200 INFO:teuthology.orchestra.run.smithi188.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T03:45:23.202 INFO:teuthology.orchestra.run.smithi188.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T03:45:23.210 INFO:teuthology.orchestra.run.smithi188.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T03:45:23.210 INFO:teuthology.orchestra.run.smithi188.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T03:45:23.215 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T03:45:23.215 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T03:45:23.216 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T03:45:23.217 INFO:teuthology.orchestra.run.smithi188.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T03:45:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T03:45:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T03:45:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T03:45:23.223 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T03:45:23.223 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T03:45:23.225 INFO:teuthology.orchestra.run.smithi188.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T03:45:23.228 INFO:teuthology.orchestra.run.smithi188.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T03:45:23.231 INFO:teuthology.orchestra.run.smithi188.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T03:45:23.232 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T03:45:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T03:45:23.232 INFO:teuthology.orchestra.run.smithi188.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T03:45:23.232 INFO:teuthology.orchestra.run.smithi188.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T03:45:23.233 INFO:teuthology.orchestra.run.smithi188.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T03:45:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T03:45:23.237 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T03:45:23.241 INFO:teuthology.orchestra.run.smithi188.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T03:45:23.242 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T03:45:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T03:45:23.245 INFO:teuthology.orchestra.run.smithi188.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T03:45:23.245 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T03:45:23.246 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T03:45:23.246 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T03:45:23.247 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T03:45:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T03:45:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T03:45:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T03:45:23.251 INFO:teuthology.orchestra.run.smithi188.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T03:45:23.251 INFO:teuthology.orchestra.run.smithi188.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T03:45:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T03:45:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T03:45:23.256 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T03:45:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T03:45:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T03:45:23.261 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T03:45:23.261 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T03:45:23.261 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T03:45:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T03:45:23.290 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T03:45:23.297 INFO:teuthology.orchestra.run.smithi188.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T03:45:23.302 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T03:45:23.322 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T03:45:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T03:45:23.326 INFO:teuthology.orchestra.run.smithi188.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T03:45:23.326 INFO:teuthology.orchestra.run.smithi188.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T03:45:23.327 INFO:teuthology.orchestra.run.smithi188.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T03:45:23.328 INFO:teuthology.orchestra.run.smithi188.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T03:45:23.329 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T03:45:23.329 INFO:teuthology.orchestra.run.smithi188.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T03:45:23.330 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T03:45:23.330 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T03:45:23.331 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T03:45:23.332 INFO:teuthology.orchestra.run.smithi188.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T03:45:23.332 INFO:teuthology.orchestra.run.smithi188.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T03:45:23.333 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T03:45:23.333 INFO:teuthology.orchestra.run.smithi188.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T03:45:23.340 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T03:45:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T03:45:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T03:45:23.344 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T03:45:23.344 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T03:45:23.371 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T03:45:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T03:45:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T03:45:23.410 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T03:45:23.411 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T03:45:23.411 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T03:45:23.412 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T03:45:23.412 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T03:45:23.413 INFO:teuthology.orchestra.run.smithi188.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T03:45:23.414 INFO:teuthology.orchestra.run.smithi188.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T03:45:28.785 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 16.0 MB in 2s (7,290 kB/s) 2024-09-16T03:45:29.260 INFO:teuthology.orchestra.run.smithi188.stdout:Fetched 16.0 MB in 3s (5,920 kB/s) 2024-09-16T03:45:29.776 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 16.0 MB in 3s (5,648 kB/s) 2024-09-16T03:45:29.830 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 16.0 MB in 3s (5,596 kB/s) 2024-09-16T03:45:30.394 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-09-16T03:45:30.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:45:30.481 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-09-16T03:45:30.692 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2024-09-16T03:45:30.692 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2024-09-16T03:45:30.696 INFO:teuthology.orchestra.run.smithi188.stdout:Reading package lists... 2024-09-16T03:45:30.715 DEBUG:teuthology.orchestra.run.smithi188:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:45:30.777 INFO:teuthology.orchestra.run.smithi188.stdout:Reading package lists... 2024-09-16T03:45:30.881 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:45:30.882 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:45:30.882 INFO:teuthology.orchestra.run.smithi117.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:45:30.882 INFO:teuthology.orchestra.run.smithi117.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:45:30.883 INFO:teuthology.orchestra.run.smithi117.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T03:45:30.883 INFO:teuthology.orchestra.run.smithi117.stdout: python2.7-minimal 2024-09-16T03:45:30.883 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:45:30.884 INFO:teuthology.orchestra.run.smithi117.stdout:The following additional packages will be installed: 2024-09-16T03:45:30.884 INFO:teuthology.orchestra.run.smithi117.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:45:30.884 INFO:teuthology.orchestra.run.smithi117.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T03:45:30.884 INFO:teuthology.orchestra.run.smithi117.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T03:45:30.886 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2024-09-16T03:45:30.886 INFO:teuthology.orchestra.run.smithi117.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T03:45:30.886 INFO:teuthology.orchestra.run.smithi117.stdout:Recommended packages: 2024-09-16T03:45:30.886 INFO:teuthology.orchestra.run.smithi117.stdout: thermald 2024-09-16T03:45:30.935 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2024-09-16T03:45:30.935 INFO:teuthology.orchestra.run.smithi117.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:45:30.935 INFO:teuthology.orchestra.run.smithi117.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T03:45:30.935 INFO:teuthology.orchestra.run.smithi117.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T03:45:30.936 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages will be upgraded: 2024-09-16T03:45:30.937 INFO:teuthology.orchestra.run.smithi117.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T03:45:30.987 INFO:teuthology.orchestra.run.smithi188.stdout:Building dependency tree... 2024-09-16T03:45:30.987 INFO:teuthology.orchestra.run.smithi117.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:45:30.987 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 113 MB of archives. 2024-09-16T03:45:30.987 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T03:45:30.987 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T03:45:30.988 INFO:teuthology.orchestra.run.smithi188.stdout:Reading state information... 2024-09-16T03:45:31.162 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:45:31.162 INFO:teuthology.orchestra.run.smithi188.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:45:31.162 INFO:teuthology.orchestra.run.smithi188.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:45:31.163 INFO:teuthology.orchestra.run.smithi188.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:45:31.163 INFO:teuthology.orchestra.run.smithi188.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T03:45:31.163 INFO:teuthology.orchestra.run.smithi188.stdout: python2.7-minimal 2024-09-16T03:45:31.163 INFO:teuthology.orchestra.run.smithi188.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:45:31.164 INFO:teuthology.orchestra.run.smithi188.stdout:The following additional packages will be installed: 2024-09-16T03:45:31.164 INFO:teuthology.orchestra.run.smithi188.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:45:31.164 INFO:teuthology.orchestra.run.smithi188.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T03:45:31.165 INFO:teuthology.orchestra.run.smithi188.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T03:45:31.166 INFO:teuthology.orchestra.run.smithi188.stdout:Suggested packages: 2024-09-16T03:45:31.166 INFO:teuthology.orchestra.run.smithi188.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T03:45:31.166 INFO:teuthology.orchestra.run.smithi188.stdout:Recommended packages: 2024-09-16T03:45:31.166 INFO:teuthology.orchestra.run.smithi188.stdout: thermald 2024-09-16T03:45:31.213 INFO:teuthology.orchestra.run.smithi188.stdout:The following NEW packages will be installed: 2024-09-16T03:45:31.213 INFO:teuthology.orchestra.run.smithi188.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:45:31.213 INFO:teuthology.orchestra.run.smithi188.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T03:45:31.214 INFO:teuthology.orchestra.run.smithi188.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T03:45:31.215 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages will be upgraded: 2024-09-16T03:45:31.216 INFO:teuthology.orchestra.run.smithi188.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T03:45:31.232 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T03:45:31.251 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:45:31.266 INFO:teuthology.orchestra.run.smithi188.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:45:31.266 INFO:teuthology.orchestra.run.smithi188.stdout:Need to get 113 MB of archives. 2024-09-16T03:45:31.266 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T03:45:31.266 INFO:teuthology.orchestra.run.smithi188.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T03:45:31.321 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T03:45:31.399 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T03:45:31.489 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T03:45:31.533 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-16T03:45:31.534 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-16T03:45:31.609 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T03:45:31.628 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:45:31.692 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T03:45:31.725 INFO:teuthology.orchestra.run.smithi188.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T03:45:31.734 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:45:31.735 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:45:31.735 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:45:31.735 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:45:31.735 INFO:teuthology.orchestra.run.smithi039.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T03:45:31.736 INFO:teuthology.orchestra.run.smithi039.stdout: python2.7-minimal 2024-09-16T03:45:31.736 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:45:31.737 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-16T03:45:31.737 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:45:31.737 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T03:45:31.737 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T03:45:31.739 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-16T03:45:31.739 INFO:teuthology.orchestra.run.smithi039.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T03:45:31.739 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-16T03:45:31.739 INFO:teuthology.orchestra.run.smithi039.stdout: thermald 2024-09-16T03:45:31.781 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-16T03:45:31.781 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:45:31.781 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T03:45:31.781 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T03:45:31.783 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages will be upgraded: 2024-09-16T03:45:31.783 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T03:45:31.814 INFO:teuthology.orchestra.run.smithi188.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T03:45:31.906 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-09-16T03:45:31.906 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-09-16T03:45:31.964 INFO:teuthology.orchestra.run.smithi039.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:45:31.964 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 113 MB of archives. 2024-09-16T03:45:31.965 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T03:45:31.965 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T03:45:32.100 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:45:32.101 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:45:32.101 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:45:32.101 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:45:32.101 INFO:teuthology.orchestra.run.smithi037.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T03:45:32.102 INFO:teuthology.orchestra.run.smithi037.stdout: python2.7-minimal 2024-09-16T03:45:32.102 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:45:32.103 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-09-16T03:45:32.103 INFO:teuthology.orchestra.run.smithi037.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:45:32.103 INFO:teuthology.orchestra.run.smithi037.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T03:45:32.103 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T03:45:32.105 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-09-16T03:45:32.105 INFO:teuthology.orchestra.run.smithi037.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T03:45:32.105 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-09-16T03:45:32.105 INFO:teuthology.orchestra.run.smithi037.stdout: thermald 2024-09-16T03:45:32.147 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-09-16T03:45:32.147 INFO:teuthology.orchestra.run.smithi037.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:45:32.147 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T03:45:32.147 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T03:45:32.149 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages will be upgraded: 2024-09-16T03:45:32.149 INFO:teuthology.orchestra.run.smithi037.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T03:45:32.204 INFO:teuthology.orchestra.run.smithi037.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:45:32.204 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 113 MB of archives. 2024-09-16T03:45:32.204 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T03:45:32.204 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T03:45:32.330 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T03:45:32.330 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T03:45:32.330 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T03:45:32.457 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T03:45:32.478 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T03:45:32.610 INFO:teuthology.orchestra.run.smithi188.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T03:45:32.610 INFO:teuthology.orchestra.run.smithi188.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T03:45:32.610 INFO:teuthology.orchestra.run.smithi188.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T03:45:32.653 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T03:45:32.727 INFO:teuthology.orchestra.run.smithi188.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T03:45:32.749 INFO:teuthology.orchestra.run.smithi188.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T03:45:32.761 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T03:45:32.949 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 113 MB in 2s (70.6 MB/s) 2024-09-16T03:45:33.091 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T03:45:33.139 INFO:teuthology.orchestra.run.smithi188.stdout:Fetched 113 MB in 2s (74.3 MB/s) 2024-09-16T03:45:33.356 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T03:45:33.388 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T03:45:33.576 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T03:45:33.576 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T03:45:33.577 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T03:45:33.707 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T03:45:33.729 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T03:45:33.764 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T03:45:34.156 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 113 MB in 2s (72.1 MB/s) 2024-09-16T03:45:34.335 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T03:45:35.087 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T03:45:35.091 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:35.213 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:35.388 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T03:45:35.392 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:35.461 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:35.919 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T03:45:35.920 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T03:45:35.920 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T03:45:36.310 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T03:45:36.364 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T03:45:36.369 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:36.474 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T03:45:36.498 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:36.904 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 113 MB in 5s (24.2 MB/s) 2024-09-16T03:45:37.106 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T03:45:37.489 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T03:45:37.510 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:37.513 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T03:45:37.534 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:37.587 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:37.629 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:38.024 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T03:45:38.045 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:38.067 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T03:45:38.074 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:38.088 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:38.125 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:38.869 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T03:45:38.892 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:39.055 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T03:45:39.059 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:39.128 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:39.145 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:39.761 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T03:45:39.782 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:39.818 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:41.505 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T03:45:41.526 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:41.613 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:42.074 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T03:45:42.095 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:42.132 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:44.685 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:44.812 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:45.128 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:45.148 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:45.224 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:45.309 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:45.502 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T03:45:45.524 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T03:45:45.568 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:45:45.651 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:45.821 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:46.124 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T03:45:46.146 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T03:45:46.182 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:45:46.561 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:46.686 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:47.044 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:47.206 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:47.550 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T03:45:47.573 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T03:45:47.617 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:45:49.172 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:49.307 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:49.675 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:49.802 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:50.106 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T03:45:50.128 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T03:45:50.163 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:45:51.380 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T03:45:51.417 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:51.455 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:52.438 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T03:45:52.467 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:52.504 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:53.459 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T03:45:53.497 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:53.534 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:54.096 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:54.218 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:54.552 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:45:54.673 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:54.799 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T03:45:54.927 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:55.389 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:55.539 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:55.710 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T03:45:55.747 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:45:55.785 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:55.906 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:45:56.027 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:56.170 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T03:45:56.254 INFO:teuthology.orchestra.run.smithi188.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:45:56.254 INFO:teuthology.orchestra.run.smithi188.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T03:45:56.254 INFO:teuthology.orchestra.run.smithi188.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:45:56.254 INFO:teuthology.orchestra.run.smithi188.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T03:45:56.289 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:56.422 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:56.839 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:45:56.985 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:45:57.344 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:45:57.496 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:57.763 INFO:teuthology.orchestra.run.smithi117.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:45:57.763 INFO:teuthology.orchestra.run.smithi117.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T03:45:57.764 INFO:teuthology.orchestra.run.smithi117.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:45:57.764 INFO:teuthology.orchestra.run.smithi117.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T03:45:58.290 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:58.341 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T03:45:58.476 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:58.761 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:59.708 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T03:45:59.834 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T03:45:59.868 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:45:59.961 INFO:teuthology.orchestra.run.smithi188.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:00.024 INFO:teuthology.orchestra.run.smithi188.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T03:46:00.025 INFO:teuthology.orchestra.run.smithi188.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:00.251 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:46:00.264 INFO:teuthology.orchestra.run.smithi037.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:00.264 INFO:teuthology.orchestra.run.smithi037.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:00.264 INFO:teuthology.orchestra.run.smithi037.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:00.264 INFO:teuthology.orchestra.run.smithi037.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:00.373 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:46:00.449 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:00.741 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:46:00.837 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:00.947 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T03:46:01.074 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:01.294 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T03:46:01.454 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T03:46:01.579 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:01.649 INFO:teuthology.orchestra.run.smithi117.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T03:46:01.649 INFO:teuthology.orchestra.run.smithi117.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:01.883 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:02.453 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:02.453 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:02.453 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:02.453 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:02.628 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:03.293 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T03:46:03.427 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T03:46:03.579 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:03.634 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T03:46:03.634 INFO:teuthology.orchestra.run.smithi037.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:04.087 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:05.514 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T03:46:05.640 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T03:46:05.766 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:46:05.835 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T03:46:05.835 INFO:teuthology.orchestra.run.smithi039.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:14.474 INFO:teuthology.orchestra.run.smithi188.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T03:46:14.475 INFO:teuthology.orchestra.run.smithi188.stdout:Sourcing file `/etc/default/grub' 2024-09-16T03:46:14.492 INFO:teuthology.orchestra.run.smithi188.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:46:14.520 INFO:teuthology.orchestra.run.smithi188.stdout:Generating grub configuration file ... 2024-09-16T03:46:14.879 INFO:teuthology.orchestra.run.smithi188.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:14.905 INFO:teuthology.orchestra.run.smithi188.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:15.292 INFO:teuthology.orchestra.run.smithi188.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:15.298 INFO:teuthology.orchestra.run.smithi188.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:15.368 INFO:teuthology.orchestra.run.smithi188.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:46:15.374 INFO:teuthology.orchestra.run.smithi188.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:46:15.526 INFO:teuthology.orchestra.run.smithi188.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:46:15.526 INFO:teuthology.orchestra.run.smithi188.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:46:15.526 INFO:teuthology.orchestra.run.smithi188.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:46:15.561 INFO:teuthology.orchestra.run.smithi188.stdout:done 2024-09-16T03:46:16.033 DEBUG:teuthology.orchestra.run.smithi188:> dpkg -s linux-image-generic 2024-09-16T03:46:16.047 INFO:teuthology.orchestra.run.smithi117.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T03:46:16.047 INFO:teuthology.orchestra.run.smithi117.stdout:Sourcing file `/etc/default/grub' 2024-09-16T03:46:16.057 INFO:teuthology.orchestra.run.smithi188.stdout:Package: linux-image-generic 2024-09-16T03:46:16.057 INFO:teuthology.orchestra.run.smithi188.stdout:Status: install ok installed 2024-09-16T03:46:16.057 INFO:teuthology.orchestra.run.smithi188.stdout:Priority: optional 2024-09-16T03:46:16.058 INFO:teuthology.orchestra.run.smithi188.stdout:Section: kernel 2024-09-16T03:46:16.058 INFO:teuthology.orchestra.run.smithi188.stdout:Installed-Size: 21 2024-09-16T03:46:16.058 INFO:teuthology.orchestra.run.smithi188.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:46:16.058 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: amd64 2024-09-16T03:46:16.058 INFO:teuthology.orchestra.run.smithi188.stdout:Source: linux-meta 2024-09-16T03:46:16.058 INFO:teuthology.orchestra.run.smithi188.stdout:Version: 5.15.0.121.121 2024-09-16T03:46:16.058 INFO:teuthology.orchestra.run.smithi188.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:46:16.059 INFO:teuthology.orchestra.run.smithi188.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:46:16.059 INFO:teuthology.orchestra.run.smithi188.stdout:Recommends: thermald 2024-09-16T03:46:16.059 INFO:teuthology.orchestra.run.smithi188.stdout:Description: Generic Linux kernel image 2024-09-16T03:46:16.059 INFO:teuthology.orchestra.run.smithi188.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:46:16.059 INFO:teuthology.orchestra.run.smithi188.stdout: available. 2024-09-16T03:46:16.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T03:46:16.060 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:46:16.060 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:46:16.060 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-16T03:46:16.060 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:46:16.060 DEBUG:teuthology.orchestra.run.smithi188:> sudo apt-get clean 2024-09-16T03:46:16.061 INFO:teuthology.orchestra.run.smithi117.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:46:16.086 INFO:teuthology.orchestra.run.smithi117.stdout:Generating grub configuration file ... 2024-09-16T03:46:16.235 DEBUG:teuthology.orchestra.run.smithi188:> sudo apt-get update 2024-09-16T03:46:16.417 INFO:teuthology.orchestra.run.smithi117.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:16.440 INFO:teuthology.orchestra.run.smithi117.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:16.480 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:46:16.480 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:46:16.550 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:46:16.631 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:46:16.798 INFO:teuthology.orchestra.run.smithi117.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:16.804 INFO:teuthology.orchestra.run.smithi117.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:16.871 INFO:teuthology.orchestra.run.smithi117.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:46:16.876 INFO:teuthology.orchestra.run.smithi117.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:46:17.041 INFO:teuthology.orchestra.run.smithi117.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:46:17.041 INFO:teuthology.orchestra.run.smithi117.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:46:17.041 INFO:teuthology.orchestra.run.smithi117.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:46:17.090 INFO:teuthology.orchestra.run.smithi117.stdout:done 2024-09-16T03:46:17.593 DEBUG:teuthology.orchestra.run.smithi117:> dpkg -s linux-image-generic 2024-09-16T03:46:17.610 INFO:teuthology.orchestra.run.smithi117.stdout:Package: linux-image-generic 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Status: install ok installed 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Priority: optional 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Section: kernel 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Installed-Size: 21 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: amd64 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Source: linux-meta 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Version: 5.15.0.121.121 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Recommends: thermald 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout:Description: Generic Linux kernel image 2024-09-16T03:46:17.611 INFO:teuthology.orchestra.run.smithi117.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:46:17.612 INFO:teuthology.orchestra.run.smithi117.stdout: available. 2024-09-16T03:46:17.612 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T03:46:17.612 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:46:17.612 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:46:17.612 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-16T03:46:17.612 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:46:17.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get clean 2024-09-16T03:46:17.746 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2024-09-16T03:46:17.870 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:46:17.947 INFO:teuthology.orchestra.run.smithi188.stdout:Reading package lists... 2024-09-16T03:46:17.965 DEBUG:teuthology.orchestra.run.smithi188:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:46:17.991 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:46:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:Reading package lists... 2024-09-16T03:46:18.094 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:46:18.153 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:46:18.254 INFO:teuthology.orchestra.run.smithi188.stdout:Building dependency tree... 2024-09-16T03:46:18.254 INFO:teuthology.orchestra.run.smithi188.stdout:Reading state information... 2024-09-16T03:46:18.447 INFO:teuthology.orchestra.run.smithi188.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T03:46:18.448 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:46:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:46:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:46:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:46:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:46:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:46:18.449 INFO:teuthology.orchestra.run.smithi188.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:46:18.492 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:46:18.494 DEBUG:teuthology.orchestra.run.smithi188:> dpkg -s linux-image-generic 2024-09-16T03:46:18.510 INFO:teuthology.orchestra.run.smithi188.stdout:Package: linux-image-generic 2024-09-16T03:46:18.510 INFO:teuthology.orchestra.run.smithi188.stdout:Status: install ok installed 2024-09-16T03:46:18.510 INFO:teuthology.orchestra.run.smithi188.stdout:Priority: optional 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Section: kernel 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Installed-Size: 21 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: amd64 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Source: linux-meta 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Version: 5.15.0.121.121 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Recommends: thermald 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout:Description: Generic Linux kernel image 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:46:18.511 INFO:teuthology.orchestra.run.smithi188.stdout: available. 2024-09-16T03:46:18.512 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-09-16T03:46:18.557 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.EtYITU9rd1 2024-09-16T03:46:18.558 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub/grub.cfg /tmp/tmp.EtYITU9rd1 2024-09-16T03:46:18.613 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.EtYITU9rd1 2024-09-16T03:46:18.737 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.EtYITU9rd1 is 10KB 2024-09-16T03:46:18.752 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.EtYITU9rd1 2024-09-16T03:46:18.757 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T03:46:18.813 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:46:18.814 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T03:46:18.814 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T03:46:18.911 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T03:46:18.911 DEBUG:teuthology.orchestra.run.smithi188:> sudo update-grub 2024-09-16T03:46:19.043 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T03:46:19.043 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub' 2024-09-16T03:46:19.064 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:46:19.092 INFO:teuthology.orchestra.run.smithi037.stdout:Generating grub configuration file ... 2024-09-16T03:46:19.327 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-09-16T03:46:19.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:46:19.398 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-09-16T03:46:19.465 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:19.487 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:19.595 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2024-09-16T03:46:19.595 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2024-09-16T03:46:19.764 INFO:teuthology.orchestra.run.smithi117.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T03:46:19.764 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:46:19.764 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:46:19.765 INFO:teuthology.orchestra.run.smithi117.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:46:19.765 INFO:teuthology.orchestra.run.smithi117.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:46:19.765 INFO:teuthology.orchestra.run.smithi117.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:46:19.765 INFO:teuthology.orchestra.run.smithi117.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:46:19.765 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:46:19.805 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:46:19.806 DEBUG:teuthology.orchestra.run.smithi117:> dpkg -s linux-image-generic 2024-09-16T03:46:19.848 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:19.853 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:19.865 INFO:teuthology.orchestra.run.smithi117.stdout:Package: linux-image-generic 2024-09-16T03:46:19.865 INFO:teuthology.orchestra.run.smithi117.stdout:Status: install ok installed 2024-09-16T03:46:19.865 INFO:teuthology.orchestra.run.smithi117.stdout:Priority: optional 2024-09-16T03:46:19.865 INFO:teuthology.orchestra.run.smithi117.stdout:Section: kernel 2024-09-16T03:46:19.865 INFO:teuthology.orchestra.run.smithi117.stdout:Installed-Size: 21 2024-09-16T03:46:19.865 INFO:teuthology.orchestra.run.smithi117.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:46:19.865 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: amd64 2024-09-16T03:46:19.865 INFO:teuthology.orchestra.run.smithi117.stdout:Source: linux-meta 2024-09-16T03:46:19.866 INFO:teuthology.orchestra.run.smithi117.stdout:Version: 5.15.0.121.121 2024-09-16T03:46:19.866 INFO:teuthology.orchestra.run.smithi117.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:46:19.866 INFO:teuthology.orchestra.run.smithi117.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:46:19.866 INFO:teuthology.orchestra.run.smithi117.stdout:Recommends: thermald 2024-09-16T03:46:19.866 INFO:teuthology.orchestra.run.smithi117.stdout:Description: Generic Linux kernel image 2024-09-16T03:46:19.866 INFO:teuthology.orchestra.run.smithi117.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:46:19.866 INFO:teuthology.orchestra.run.smithi117.stdout: available. 2024-09-16T03:46:19.866 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.KRaSTtyPK8 2024-09-16T03:46:19.911 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub/grub.cfg /tmp/tmp.KRaSTtyPK8 2024-09-16T03:46:19.919 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:46:19.924 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:46:19.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.KRaSTtyPK8 2024-09-16T03:46:20.083 INFO:teuthology.orchestra.run.smithi037.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:46:20.083 INFO:teuthology.orchestra.run.smithi037.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:46:20.083 INFO:teuthology.orchestra.run.smithi037.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:46:20.086 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.KRaSTtyPK8 is 10KB 2024-09-16T03:46:20.098 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.KRaSTtyPK8 2024-09-16T03:46:20.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T03:46:20.105 INFO:teuthology.orchestra.run.smithi188.stderr:Sourcing file `/etc/default/grub' 2024-09-16T03:46:20.106 INFO:teuthology.orchestra.run.smithi188.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:46:20.108 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-09-16T03:46:20.123 INFO:teuthology.orchestra.run.smithi037.stdout:done 2024-09-16T03:46:20.157 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:46:20.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T03:46:20.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T03:46:20.213 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T03:46:20.213 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub' 2024-09-16T03:46:20.224 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:46:20.240 INFO:teuthology.orchestra.run.smithi039.stdout:Generating grub configuration file ... 2024-09-16T03:46:20.245 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T03:46:20.245 DEBUG:teuthology.orchestra.run.smithi117:> sudo update-grub 2024-09-16T03:46:20.357 INFO:teuthology.orchestra.run.smithi188.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:20.363 INFO:teuthology.orchestra.run.smithi188.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:20.524 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:20.552 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:20.652 INFO:teuthology.orchestra.run.smithi188.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:20.657 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic 2024-09-16T03:46:20.658 INFO:teuthology.orchestra.run.smithi188.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:20.678 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic 2024-09-16T03:46:20.678 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2024-09-16T03:46:20.678 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 21 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.15.0.121.121 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2024-09-16T03:46:20.679 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:46:20.680 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2024-09-16T03:46:20.680 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T03:46:20.680 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:46:20.680 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:46:20.680 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T03:46:20.680 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:46:20.680 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2024-09-16T03:46:20.699 INFO:teuthology.orchestra.run.smithi188.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:46:20.704 INFO:teuthology.orchestra.run.smithi188.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:46:20.844 INFO:teuthology.orchestra.run.smithi188.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:46:20.844 INFO:teuthology.orchestra.run.smithi188.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:46:20.844 INFO:teuthology.orchestra.run.smithi188.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:46:20.856 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-09-16T03:46:20.872 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-09-16T03:46:20.914 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:20.919 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:20.974 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-09-16T03:46:20.988 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:46:20.994 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:46:21.096 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:46:21.155 INFO:teuthology.orchestra.run.smithi039.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:46:21.155 INFO:teuthology.orchestra.run.smithi039.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:46:21.155 INFO:teuthology.orchestra.run.smithi039.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:46:21.193 INFO:teuthology.orchestra.run.smithi039.stdout:done 2024-09-16T03:46:21.218 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:46:21.298 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:46:21.380 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:46:21.638 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-16T03:46:21.652 INFO:teuthology.orchestra.run.smithi117.stderr:Sourcing file `/etc/default/grub' 2024-09-16T03:46:21.654 INFO:teuthology.orchestra.run.smithi117.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:46:21.656 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-16T03:46:21.657 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-16T03:46:21.657 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-16T03:46:21.657 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-16T03:46:21.657 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-16T03:46:21.657 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:46:21.658 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-16T03:46:21.658 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-16T03:46:21.658 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.121.121 2024-09-16T03:46:21.658 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:46:21.658 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:46:21.659 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-16T03:46:21.659 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-16T03:46:21.659 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:46:21.659 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-16T03:46:21.659 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T03:46:21.660 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:46:21.660 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:46:21.660 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T03:46:21.660 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:46:21.661 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-16T03:46:21.663 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-09-16T03:46:21.833 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-16T03:46:21.837 INFO:teuthology.orchestra.run.smithi117.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:21.843 INFO:teuthology.orchestra.run.smithi117.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:21.958 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:46:21.958 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:46:21.966 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:46:21.983 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:46:22.102 INFO:teuthology.orchestra.run.smithi117.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:22.107 INFO:teuthology.orchestra.run.smithi117.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:22.143 INFO:teuthology.orchestra.run.smithi117.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:46:22.147 INFO:teuthology.orchestra.run.smithi117.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:46:22.254 INFO:teuthology.orchestra.run.smithi117.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:46:22.255 INFO:teuthology.orchestra.run.smithi117.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:46:22.255 INFO:teuthology.orchestra.run.smithi117.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:46:22.264 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-09-16T03:46:22.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-09-16T03:46:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T03:46:22.847 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:46:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T03:46:23.127 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-09-16T03:46:23.127 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-09-16T03:46:23.330 INFO:teuthology.orchestra.run.smithi037.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T03:46:23.330 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:46:23.330 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:46:23.330 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:46:23.330 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:46:23.331 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:46:23.331 INFO:teuthology.orchestra.run.smithi037.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:46:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:46:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:46:23.377 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic 2024-09-16T03:46:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic 2024-09-16T03:46:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2024-09-16T03:46:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 21 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.15.0.121.121 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:46:23.395 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2024-09-16T03:46:23.396 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-09-16T03:46:23.441 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T03:46:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.VXXEmYMVwu 2024-09-16T03:46:23.443 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub/grub.cfg /tmp/tmp.VXXEmYMVwu 2024-09-16T03:46:23.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:46:23.499 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.VXXEmYMVwu 2024-09-16T03:46:23.525 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T03:46:23.614 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.VXXEmYMVwu is 10KB 2024-09-16T03:46:23.626 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.VXXEmYMVwu 2024-09-16T03:46:23.632 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T03:46:23.687 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:46:23.687 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T03:46:23.687 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T03:46:23.751 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-16T03:46:23.752 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-16T03:46:23.772 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T03:46:23.772 DEBUG:teuthology.orchestra.run.smithi037:> sudo update-grub 2024-09-16T03:46:23.953 INFO:teuthology.orchestra.run.smithi039.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T03:46:23.954 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:46:23.954 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:46:23.954 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:46:23.954 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:46:23.955 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:46:23.955 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:46:23.955 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:46:23.998 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:46:24.000 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-16T03:46:24.016 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-16T03:46:24.016 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.121.121 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:46:24.017 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-16T03:46:24.018 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-16T03:46:24.067 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.spkTNjnJdh 2024-09-16T03:46:24.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub/grub.cfg /tmp/tmp.spkTNjnJdh 2024-09-16T03:46:24.124 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.spkTNjnJdh 2024-09-16T03:46:24.249 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.spkTNjnJdh is 10KB 2024-09-16T03:46:24.301 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.spkTNjnJdh 2024-09-16T03:46:24.308 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T03:46:24.363 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:46:24.364 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T03:46:24.364 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T03:46:24.431 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T03:46:24.431 DEBUG:teuthology.orchestra.run.smithi039:> sudo update-grub 2024-09-16T03:46:25.154 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub' 2024-09-16T03:46:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:46:25.160 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-09-16T03:46:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:25.363 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:25.636 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:25.641 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:25.684 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:46:25.689 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:46:25.712 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub' 2024-09-16T03:46:25.712 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:46:25.714 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-09-16T03:46:25.889 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:46:25.895 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:46:25.931 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:46:25.931 INFO:teuthology.orchestra.run.smithi037.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:46:25.931 INFO:teuthology.orchestra.run.smithi037.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:46:25.958 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-09-16T03:46:25.967 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-09-16T03:46:26.159 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:46:26.164 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:46:26.201 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:46:26.206 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:46:26.328 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:46:26.328 INFO:teuthology.orchestra.run.smithi039.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:46:26.328 INFO:teuthology.orchestra.run.smithi039.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:46:26.345 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-09-16T03:46:26.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-09-16T03:46:50.899 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-16T03:46:50.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-16T03:46:50.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:46:52.269 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-16T03:46:52.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-16T03:46:52.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:46:55.973 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T03:46:55.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T03:46:55.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:46:56.357 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T03:46:56.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T03:46:56.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:47:26.909 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-09-16T03:47:31.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-16T03:47:35.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-16T03:47:35.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:47:40.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T03:47:40.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:47:43.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-16T03:47:50.911 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:47:54.301 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-09-16T03:47:55.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T03:47:55.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:47:55.977 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:47:58.141 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-16T03:47:59.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-16T03:47:59.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:47:59.915 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-09-16T03:48:04.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T03:48:04.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:48:04.983 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-09-16T03:48:06.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-16T03:48:06.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:48:06.802 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-09-16T03:48:07.718 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-16T03:48:07.718 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-121-generic"... 2024-09-16T03:48:07.719 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-09-16T03:48:07.766 INFO:teuthology.orchestra.run.smithi117.stdout:5.15.0-121-generic 2024-09-16T03:48:07.767 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T03:48:07.767 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:48:07.767 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T03:48:08.767 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-16T03:48:08.768 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:48:08.869 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-09-16T03:48:08.900 DEBUG:teuthology.parallel:result is None 2024-09-16T03:48:11.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-16T03:48:11.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:48:12.227 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-09-16T03:48:12.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-16T03:48:12.726 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-121-generic"... 2024-09-16T03:48:12.726 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-09-16T03:48:12.773 INFO:teuthology.orchestra.run.smithi188.stdout:5.15.0-121-generic 2024-09-16T03:48:12.773 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T03:48:12.773 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:48:12.773 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T03:48:13.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T03:48:13.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:48:13.775 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-16T03:48:13.775 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:48:14.147 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-09-16T03:48:14.178 DEBUG:teuthology.parallel:result is None 2024-09-16T03:48:16.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T03:48:16.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:48:17.185 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-16T03:48:17.812 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T03:48:17.813 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T03:48:17.813 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-16T03:48:17.858 INFO:teuthology.orchestra.run.smithi037.stdout:5.15.0-121-generic 2024-09-16T03:48:17.858 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T03:48:17.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:48:17.858 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T03:48:18.859 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T03:48:18.859 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:48:18.936 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-09-16T03:48:18.968 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:13.145 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:49:31.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T03:49:31.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:49:31.402 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-16T03:49:32.219 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T03:49:32.219 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T03:49:32.219 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-16T03:49:32.267 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-121-generic 2024-09-16T03:49:32.268 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T03:49:32.268 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:49:32.268 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T03:49:33.269 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T03:49:33.269 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:49:33.337 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-09-16T03:49:33.361 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:33.361 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T03:49:33.371 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T03:49:33.371 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:49:33.374 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:49:33.376 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:49:33.379 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:49:33.386 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T03:49:33.420 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T03:49:33.434 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T03:49:33.434 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:49:33.436 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:49:33.439 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:49:33.441 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:49:33.506 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T03:49:33.518 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T03:49:33.519 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:49:33.522 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:49:33.525 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:49:33.527 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:49:33.543 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:49:33.545 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:49:33.547 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:49:33.549 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:49:33.552 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:49:33.559 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:49:33.559 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:49:33.561 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:49:33.561 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T03:49:33.572 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T03:49:33.573 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:49:33.607 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:49:33.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:49:33.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:49:33.628 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T03:49:33.637 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T03:49:33.637 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:49:33.671 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:49:33.673 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:49:33.676 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:49:33.682 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:49:33.726 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:49:33.774 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:49:33.774 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:49:33.837 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:49:33.844 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:49:33.892 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:49:33.892 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:49:33.961 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:49:33.967 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:49:34.014 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:49:34.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:49:34.083 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:49:34.089 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:49:34.137 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:49:34.137 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:49:34.201 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-09-16T03:49:34.203 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-09-16T03:49:34.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-09-16T03:49:34.208 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-09-16T03:49:34.275 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T03:49:34.284 INFO:teuthology.task.internal:Starting timer... 2024-09-16T03:49:34.284 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T03:49:34.301 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T03:49:34.316 DEBUG:teuthology.task.selinux:Excluding smithi037: OS 'ubuntu' does not support SELinux 2024-09-16T03:49:34.316 DEBUG:teuthology.task.selinux:Excluding smithi039: OS 'ubuntu' does not support SELinux 2024-09-16T03:49:34.317 DEBUG:teuthology.task.selinux:Excluding smithi117: OS 'ubuntu' does not support SELinux 2024-09-16T03:49:34.317 DEBUG:teuthology.task.selinux:Excluding smithi188: OS 'ubuntu' does not support SELinux 2024-09-16T03:49:34.317 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T03:49:34.317 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T03:49:34.317 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T03:49:34.317 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T03:49:34.330 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T03:49:34.333 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T03:49:34.448 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T03:49:34.459 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T03:49:34.461 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi039.front.sepia.ceph.com,smithi117.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T03:57:14.213 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-09-16T03:57:14.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T03:57:14.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:57:14.307 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-16T03:57:14.389 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T03:57:14.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T03:57:14.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:57:14.480 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-16T03:57:14.561 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-16T03:57:14.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-16T03:57:14.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:57:14.643 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-09-16T03:57:14.721 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-16T03:57:14.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-16T03:57:14.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:57:14.805 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-09-16T03:57:14.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-09-16T03:57:14.886 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T03:57:14.896 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T03:57:14.897 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:57:14.897 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:57:14.901 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:57:14.901 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:57:14.904 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:57:14.905 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:57:14.908 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:57:14.908 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T03:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Command line: ntpd -gq 2024-09-16T03:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: ---------------------------------------------------- 2024-09-16T03:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T03:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T03:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: corporation. Support and training for ntp-4 are 2024-09-16T03:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: available at https://www.nwtime.org/support 2024-09-16T03:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: ---------------------------------------------------- 2024-09-16T03:57:14.930 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: proto: precision = 0.046 usec (-24) 2024-09-16T03:57:14.930 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: basedate set to 2022-02-04 2024-09-16T03:57:14.930 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: gps base set to 2022-02-06 (week 2196) 2024-09-16T03:57:14.930 INFO:teuthology.orchestra.run.smithi037.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:57:14.931 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:57:14.931 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:57:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:57:14.932 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T03:57:14.932 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T03:57:14.932 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T03:57:14.932 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Listen normally on 3 ens1f0 172.21.15.37:123 2024-09-16T03:57:14.932 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Listen normally on 4 lo [::1]:123 2024-09-16T03:57:14.932 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe88:7794%4]:123 2024-09-16T03:57:14.932 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:14 ntpd[18378]: Listening on routing socket on fd #22 for interface updates 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Command line: ntpd -gq 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: ---------------------------------------------------- 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: corporation. Support and training for ntp-4 are 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: available at https://www.nwtime.org/support 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: ---------------------------------------------------- 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: proto: precision = 0.044 usec (-24) 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: basedate set to 2022-02-04 2024-09-16T03:57:14.934 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: gps base set to 2022-02-06 (week 2196) 2024-09-16T03:57:14.935 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:57:14.935 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:57:14.935 INFO:teuthology.orchestra.run.smithi039.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:57:14.936 INFO:teuthology.orchestra.run.smithi039.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:57:14.937 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T03:57:14.937 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T03:57:14.937 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T03:57:14.937 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Listen normally on 3 ens1f0 172.21.15.39:123 2024-09-16T03:57:14.937 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Listen normally on 4 lo [::1]:123 2024-09-16T03:57:14.937 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1370%4]:123 2024-09-16T03:57:14.938 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:14 ntpd[18367]: Listening on routing socket on fd #22 for interface updates 2024-09-16T03:57:14.938 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Command line: ntpd -gq 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: ---------------------------------------------------- 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: corporation. Support and training for ntp-4 are 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: available at https://www.nwtime.org/support 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: ---------------------------------------------------- 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: proto: precision = 0.060 usec (-24) 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: basedate set to 2022-02-04 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: gps base set to 2022-02-06 (week 2196) 2024-09-16T03:57:14.939 INFO:teuthology.orchestra.run.smithi117.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:57:14.940 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:57:14.940 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:57:14.940 INFO:teuthology.orchestra.run.smithi117.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:57:14.941 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T03:57:14.941 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T03:57:14.941 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T03:57:14.941 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Listen normally on 3 enp3s0f1 172.21.15.117:123 2024-09-16T03:57:14.941 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Listen normally on 4 lo [::1]:123 2024-09-16T03:57:14.941 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4463%5]:123 2024-09-16T03:57:14.941 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:14 ntpd[18520]: Listening on routing socket on fd #22 for interface updates 2024-09-16T03:57:14.955 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T03:57:14.955 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Command line: ntpd -gq 2024-09-16T03:57:14.955 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: ---------------------------------------------------- 2024-09-16T03:57:14.955 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T03:57:14.955 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T03:57:14.956 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: corporation. Support and training for ntp-4 are 2024-09-16T03:57:14.956 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: available at https://www.nwtime.org/support 2024-09-16T03:57:14.956 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: ---------------------------------------------------- 2024-09-16T03:57:14.957 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: proto: precision = 0.055 usec (-24) 2024-09-16T03:57:14.957 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: basedate set to 2022-02-04 2024-09-16T03:57:14.958 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: gps base set to 2022-02-06 (week 2196) 2024-09-16T03:57:14.958 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:57:14.958 INFO:teuthology.orchestra.run.smithi188.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:57:14.958 INFO:teuthology.orchestra.run.smithi188.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:57:14.959 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:57:14.959 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T03:57:14.959 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T03:57:14.959 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T03:57:14.959 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Listen normally on 3 enp3s0f1 172.21.15.188:123 2024-09-16T03:57:14.959 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Listen normally on 4 lo [::1]:123 2024-09-16T03:57:14.960 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8039%5]:123 2024-09-16T03:57:14.960 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:14 ntpd[18475]: Listening on routing socket on fd #22 for interface updates 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:15.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:15.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:15.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:15.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:15 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:15.955 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:15 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:15.955 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:15 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:16.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:16 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:16.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:16 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:16.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:16 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:16.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:16 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:16.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:16 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:16.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:16 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:16.931 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:16 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:16.931 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:16 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:16.931 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:16 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:16.931 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:16 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:16.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:16 ntpd[18367]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:16.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:16 ntpd[18367]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:16.955 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:16 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:16.955 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:16 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:16.955 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:16 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:16.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:16 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:16.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:16 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:16.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:16 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:17 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:17.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:17.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:17.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:17.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:17.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:17.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:17.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:17.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:17.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:17.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:17.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:17.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:17 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:18.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:18 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:18.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:18 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:18.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:18 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:18.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:18 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:18.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:18 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:18.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:18 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:18.931 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:18 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:18.931 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:18 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:18.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:18 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:18.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:18 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:18.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:18 ntpd[18367]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:18.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:18 ntpd[18367]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:18.934 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:18 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:18.934 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:18 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:18.955 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:18 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:18.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:18 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:18.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:18 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:18.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:18 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:18.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:18 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:18.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:18 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:19 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:19 ntpd[18367]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:19 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:19.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:19 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:20.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:20 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:20.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:20 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:20.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:20 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:20.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:20 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:20.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:20 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:20.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:20 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:20.931 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:20 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:20.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:20 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:20.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:20 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:20.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:20 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:20.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:20 ntpd[18367]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:20.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:20 ntpd[18367]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:20.955 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:20 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:20.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:20 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:20.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:20 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:20.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:20 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:20.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:20 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:20.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:20 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:21.929 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.930 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 03:57:21 ntpd[18378]: ntpd: time slew +0.000275 s 2024-09-16T03:57:21.930 INFO:teuthology.orchestra.run.smithi037.stdout:ntpd: time slew +0.000275s 2024-09-16T03:57:21.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T03:57:21.930 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 03:57:21 ntpd[18378]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:21.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:21 ntpd[18367]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:21.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:21.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:21 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:21.958 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:21.958 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:21.958 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:21 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:21.980 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:57:21.980 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-09-16T03:57:21.980 INFO:teuthology.orchestra.run.smithi037.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:21.980 INFO:teuthology.orchestra.run.smithi037.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:21.980 INFO:teuthology.orchestra.run.smithi037.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:21.980 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:22.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:22 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:22.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:22 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:22.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:22 ntpd[18367]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:22.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:22 ntpd[18367]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:22.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:22 ntpd[18367]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:22.932 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:22 ntpd[18367]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:22.932 INFO:teuthology.orchestra.run.smithi039.stdout:16 Sep 03:57:22 ntpd[18367]: ntpd: time slew -0.000033 s 2024-09-16T03:57:22.933 INFO:teuthology.orchestra.run.smithi039.stdout:ntpd: time slew -0.000033s 2024-09-16T03:57:22.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:22 ntpd[18367]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T03:57:22.933 INFO:teuthology.orchestra.run.smithi039.stderr:16 Sep 03:57:22 ntpd[18367]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T03:57:22.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:22 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:22.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:22 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:22.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:22 ntpd[18475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:22.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:22 ntpd[18475]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:22.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:22 ntpd[18475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:22.956 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:22 ntpd[18475]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:22.956 INFO:teuthology.orchestra.run.smithi188.stdout:16 Sep 03:57:22 ntpd[18475]: ntpd: time slew -0.000363 s 2024-09-16T03:57:22.957 INFO:teuthology.orchestra.run.smithi188.stdout:ntpd: time slew -0.000363s 2024-09-16T03:57:22.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:22 ntpd[18475]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T03:57:22.957 INFO:teuthology.orchestra.run.smithi188.stderr:16 Sep 03:57:22 ntpd[18475]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T03:57:22.982 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:57:22.982 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-16T03:57:22.982 INFO:teuthology.orchestra.run.smithi039.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:22.982 INFO:teuthology.orchestra.run.smithi039.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:22.982 INFO:teuthology.orchestra.run.smithi039.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:22.982 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:57:23.012 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================== 2024-09-16T03:57:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:23.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:23.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:23.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:23.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:23.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:23.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:23 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:25.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:25.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:25.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:25.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:25.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:25.935 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:25.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:57:25.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:57:25.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:57:25.936 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:57:25.936 INFO:teuthology.orchestra.run.smithi117.stdout:16 Sep 03:57:25 ntpd[18520]: ntpd: time slew -0.000415 s 2024-09-16T03:57:25.936 INFO:teuthology.orchestra.run.smithi117.stdout:ntpd: time slew -0.000415s 2024-09-16T03:57:25.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T03:57:25.937 INFO:teuthology.orchestra.run.smithi117.stderr:16 Sep 03:57:25 ntpd[18520]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T03:57:25.994 INFO:teuthology.orchestra.run.smithi117.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:57:25.995 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================== 2024-09-16T03:57:25.995 INFO:teuthology.orchestra.run.smithi117.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:25.995 INFO:teuthology.orchestra.run.smithi117.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:25.995 INFO:teuthology.orchestra.run.smithi117.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:25.995 INFO:teuthology.orchestra.run.smithi117.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:57:25.996 INFO:teuthology.run_tasks:Running task install... 2024-09-16T03:57:26.006 DEBUG:teuthology.task.install:project ceph 2024-09-16T03:57:26.006 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T03:57:26.006 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T03:57:26.006 INFO:teuthology.task.install:Using flavor: default 2024-09-16T03:57:26.012 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T03:57:26.012 INFO:teuthology.task.install:extra packages: [] 2024-09-16T03:57:26.016 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-key list | grep Ceph 2024-09-16T03:57:26.017 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-key list | grep Ceph 2024-09-16T03:57:26.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-key list | grep Ceph 2024-09-16T03:57:26.018 DEBUG:teuthology.orchestra.run.smithi188:> sudo apt-key list | grep Ceph 2024-09-16T03:57:26.081 INFO:teuthology.orchestra.run.smithi188.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T03:57:26.083 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T03:57:26.085 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T03:57:26.100 INFO:teuthology.orchestra.run.smithi117.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T03:57:26.109 INFO:teuthology.orchestra.run.smithi188.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T03:57:26.110 INFO:teuthology.orchestra.run.smithi188.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T03:57:26.110 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T03:57:26.110 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:57:26.116 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T03:57:26.116 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T03:57:26.116 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T03:57:26.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:57:26.121 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T03:57:26.121 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T03:57:26.121 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T03:57:26.121 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:57:26.134 INFO:teuthology.orchestra.run.smithi117.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T03:57:26.134 INFO:teuthology.orchestra.run.smithi117.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T03:57:26.134 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T03:57:26.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:57:26.273 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T03:57:26.273 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:57:26.278 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T03:57:26.278 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:57:26.279 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T03:57:26.279 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:57:26.285 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T03:57:26.286 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:57:26.388 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:57:26.388 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T03:57:26.393 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:57:26.393 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T03:57:26.397 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:57:26.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T03:57:26.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-16T03:57:26.406 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:57:26.406 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T03:57:26.407 DEBUG:teuthology.orchestra.run.smithi188:> sudo apt-get update 2024-09-16T03:57:26.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2024-09-16T03:57:26.422 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-09-16T03:57:26.574 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:57:26.576 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:57:26.577 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:57:26.587 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:57:26.596 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:57:26.604 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:57:26.606 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:57:26.624 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:57:26.630 INFO:teuthology.orchestra.run.smithi188.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T03:57:26.630 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T03:57:26.632 INFO:teuthology.orchestra.run.smithi117.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T03:57:26.651 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T03:57:26.651 INFO:teuthology.orchestra.run.smithi188.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T03:57:26.651 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T03:57:26.655 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T03:57:26.672 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T03:57:26.673 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T03:57:26.673 INFO:teuthology.orchestra.run.smithi188.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T03:57:26.677 INFO:teuthology.orchestra.run.smithi117.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T03:57:26.694 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T03:57:26.694 INFO:teuthology.orchestra.run.smithi188.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T03:57:26.694 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T03:57:26.698 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T03:57:26.699 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:57:26.700 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:57:26.703 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:57:26.715 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T03:57:26.721 INFO:teuthology.orchestra.run.smithi188.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T03:57:26.721 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:57:26.721 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T03:57:26.724 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T03:57:26.742 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T03:57:26.777 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:57:26.779 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:57:26.859 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:57:26.860 INFO:teuthology.orchestra.run.smithi188.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:57:26.915 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 29.0 kB in 0s (88.6 kB/s) 2024-09-16T03:57:26.958 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 29.0 kB in 0s (84.5 kB/s) 2024-09-16T03:57:26.999 INFO:teuthology.orchestra.run.smithi188.stdout:Fetched 29.0 kB in 0s (70.5 kB/s) 2024-09-16T03:57:27.007 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 29.0 kB in 0s (69.8 kB/s) 2024-09-16T03:57:27.924 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-09-16T03:57:27.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:57:27.965 INFO:teuthology.orchestra.run.smithi188.stdout:Reading package lists... 2024-09-16T03:57:27.982 DEBUG:teuthology.orchestra.run.smithi188:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:57:28.003 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-09-16T03:57:28.006 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T03:57:28.023 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:57:28.036 INFO:teuthology.orchestra.run.smithi188.stdout:Reading package lists... 2024-09-16T03:57:28.057 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T03:57:28.074 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:57:28.089 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T03:57:28.130 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-16T03:57:28.211 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2024-09-16T03:57:28.212 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2024-09-16T03:57:28.223 INFO:teuthology.orchestra.run.smithi188.stdout:Building dependency tree... 2024-09-16T03:57:28.224 INFO:teuthology.orchestra.run.smithi188.stdout:Reading state information... 2024-09-16T03:57:28.310 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-09-16T03:57:28.310 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-09-16T03:57:28.321 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-16T03:57:28.322 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-16T03:57:28.408 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:57:28.408 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:57:28.408 INFO:teuthology.orchestra.run.smithi117.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:57:28.409 INFO:teuthology.orchestra.run.smithi117.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout:The following additional packages will be installed: 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T03:57:28.412 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T03:57:28.413 INFO:teuthology.orchestra.run.smithi117.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T03:57:28.414 INFO:teuthology.orchestra.run.smithi117.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T03:57:28.414 INFO:teuthology.orchestra.run.smithi117.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T03:57:28.414 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T03:57:28.414 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T03:57:28.414 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet zip 2024-09-16T03:57:28.415 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:57:28.415 INFO:teuthology.orchestra.run.smithi188.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:57:28.415 INFO:teuthology.orchestra.run.smithi188.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:57:28.415 INFO:teuthology.orchestra.run.smithi188.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:57:28.415 INFO:teuthology.orchestra.run.smithi188.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:57:28.415 INFO:teuthology.orchestra.run.smithi188.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout:The following additional packages will be installed: 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T03:57:28.416 INFO:teuthology.orchestra.run.smithi188.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T03:57:28.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T03:57:28.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T03:57:28.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:57:28.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:57:28.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:57:28.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:57:28.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet zip 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout:Suggested packages: 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T03:57:28.418 INFO:teuthology.orchestra.run.smithi188.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T03:57:28.419 INFO:teuthology.orchestra.run.smithi188.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T03:57:28.419 INFO:teuthology.orchestra.run.smithi188.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T03:57:28.419 INFO:teuthology.orchestra.run.smithi188.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T03:57:28.419 INFO:teuthology.orchestra.run.smithi188.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T03:57:28.419 INFO:teuthology.orchestra.run.smithi188.stdout:Recommended packages: 2024-09-16T03:57:28.419 INFO:teuthology.orchestra.run.smithi188.stdout: btrfs-tools 2024-09-16T03:57:28.419 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2024-09-16T03:57:28.419 INFO:teuthology.orchestra.run.smithi117.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T03:57:28.420 INFO:teuthology.orchestra.run.smithi117.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T03:57:28.420 INFO:teuthology.orchestra.run.smithi117.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T03:57:28.420 INFO:teuthology.orchestra.run.smithi117.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T03:57:28.420 INFO:teuthology.orchestra.run.smithi117.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T03:57:28.420 INFO:teuthology.orchestra.run.smithi117.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T03:57:28.420 INFO:teuthology.orchestra.run.smithi117.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T03:57:28.420 INFO:teuthology.orchestra.run.smithi117.stdout:Recommended packages: 2024-09-16T03:57:28.420 INFO:teuthology.orchestra.run.smithi117.stdout: btrfs-tools 2024-09-16T03:57:28.515 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2024-09-16T03:57:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T03:57:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T03:57:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T03:57:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T03:57:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T03:57:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T03:57:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T03:57:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T03:57:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T03:57:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T03:57:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T03:57:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T03:57:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T03:57:28.521 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:57:28.521 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:57:28.521 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:57:28.521 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:57:28.521 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:57:28.521 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:57:28.521 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout:The following NEW packages will be installed: 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T03:57:28.522 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T03:57:28.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T03:57:28.524 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T03:57:28.524 INFO:teuthology.orchestra.run.smithi188.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T03:57:28.526 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-16T03:57:28.526 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T03:57:28.526 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:57:28.527 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T03:57:28.528 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T03:57:28.528 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T03:57:28.528 INFO:teuthology.orchestra.run.smithi039.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T03:57:28.528 INFO:teuthology.orchestra.run.smithi039.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T03:57:28.528 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T03:57:28.528 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T03:57:28.528 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet zip 2024-09-16T03:57:28.529 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-16T03:57:28.529 INFO:teuthology.orchestra.run.smithi039.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T03:57:28.529 INFO:teuthology.orchestra.run.smithi039.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T03:57:28.529 INFO:teuthology.orchestra.run.smithi039.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T03:57:28.529 INFO:teuthology.orchestra.run.smithi039.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T03:57:28.529 INFO:teuthology.orchestra.run.smithi039.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T03:57:28.530 INFO:teuthology.orchestra.run.smithi039.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T03:57:28.530 INFO:teuthology.orchestra.run.smithi039.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T03:57:28.530 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-16T03:57:28.530 INFO:teuthology.orchestra.run.smithi039.stdout: btrfs-tools 2024-09-16T03:57:28.542 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:57:28.542 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:57:28.542 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:57:28.542 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:57:28.543 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:57:28.543 INFO:teuthology.orchestra.run.smithi037.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:57:28.543 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:57:28.544 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-09-16T03:57:28.544 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T03:57:28.545 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T03:57:28.545 INFO:teuthology.orchestra.run.smithi037.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T03:57:28.545 INFO:teuthology.orchestra.run.smithi037.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T03:57:28.546 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T03:57:28.547 INFO:teuthology.orchestra.run.smithi037.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T03:57:28.547 INFO:teuthology.orchestra.run.smithi037.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T03:57:28.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T03:57:28.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T03:57:28.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:57:28.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:57:28.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:57:28.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:57:28.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:57:28.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:57:28.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:57:28.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T03:57:28.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T03:57:28.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T03:57:28.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T03:57:28.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T03:57:28.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T03:57:28.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T03:57:28.550 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet zip 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-09-16T03:57:28.551 INFO:teuthology.orchestra.run.smithi037.stdout: btrfs-tools 2024-09-16T03:57:28.577 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T03:57:28.577 INFO:teuthology.orchestra.run.smithi188.stdout:Need to get 181 MB of archives. 2024-09-16T03:57:28.577 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T03:57:28.577 INFO:teuthology.orchestra.run.smithi188.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T03:57:28.630 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-16T03:57:28.630 INFO:teuthology.orchestra.run.smithi039.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T03:57:28.630 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T03:57:28.630 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T03:57:28.630 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T03:57:28.630 INFO:teuthology.orchestra.run.smithi039.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T03:57:28.630 INFO:teuthology.orchestra.run.smithi039.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:57:28.632 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T03:57:28.633 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T03:57:28.633 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T03:57:28.633 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T03:57:28.633 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T03:57:28.633 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T03:57:28.633 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T03:57:28.633 INFO:teuthology.orchestra.run.smithi039.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T03:57:28.657 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-09-16T03:57:28.657 INFO:teuthology.orchestra.run.smithi037.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T03:57:28.657 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T03:57:28.658 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T03:57:28.658 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T03:57:28.658 INFO:teuthology.orchestra.run.smithi037.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T03:57:28.658 INFO:teuthology.orchestra.run.smithi037.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T03:57:28.658 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T03:57:28.658 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 181 MB of archives. 2024-09-16T03:57:28.658 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T03:57:28.659 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:57:28.660 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T03:57:28.661 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T03:57:28.661 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T03:57:28.661 INFO:teuthology.orchestra.run.smithi037.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T03:57:28.661 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T03:57:28.661 INFO:teuthology.orchestra.run.smithi037.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T03:57:28.661 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T03:57:28.661 INFO:teuthology.orchestra.run.smithi037.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T03:57:28.661 INFO:teuthology.orchestra.run.smithi188.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T03:57:28.664 INFO:teuthology.orchestra.run.smithi188.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T03:57:28.667 INFO:teuthology.orchestra.run.smithi188.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T03:57:28.684 INFO:teuthology.orchestra.run.smithi188.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T03:57:28.691 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T03:57:28.692 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 181 MB of archives. 2024-09-16T03:57:28.692 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T03:57:28.692 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T03:57:28.702 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T03:57:28.722 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T03:57:28.722 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 181 MB of archives. 2024-09-16T03:57:28.722 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T03:57:28.722 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T03:57:28.744 INFO:teuthology.orchestra.run.smithi188.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T03:57:28.750 INFO:teuthology.orchestra.run.smithi188.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T03:57:28.763 INFO:teuthology.orchestra.run.smithi188.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T03:57:28.766 INFO:teuthology.orchestra.run.smithi188.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T03:57:28.767 INFO:teuthology.orchestra.run.smithi188.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T03:57:28.767 INFO:teuthology.orchestra.run.smithi188.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T03:57:28.775 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T03:57:28.776 INFO:teuthology.orchestra.run.smithi188.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T03:57:28.778 INFO:teuthology.orchestra.run.smithi188.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T03:57:28.780 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T03:57:28.780 INFO:teuthology.orchestra.run.smithi188.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T03:57:28.781 INFO:teuthology.orchestra.run.smithi188.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T03:57:28.783 INFO:teuthology.orchestra.run.smithi188.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T03:57:28.783 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T03:57:28.785 INFO:teuthology.orchestra.run.smithi188.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T03:57:28.799 INFO:teuthology.orchestra.run.smithi188.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T03:57:28.800 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T03:57:28.801 INFO:teuthology.orchestra.run.smithi188.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T03:57:28.802 INFO:teuthology.orchestra.run.smithi188.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T03:57:28.802 INFO:teuthology.orchestra.run.smithi188.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T03:57:28.803 INFO:teuthology.orchestra.run.smithi188.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T03:57:28.803 INFO:teuthology.orchestra.run.smithi188.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T03:57:28.804 INFO:teuthology.orchestra.run.smithi188.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T03:57:28.804 INFO:teuthology.orchestra.run.smithi188.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T03:57:28.811 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T03:57:28.814 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T03:57:28.817 INFO:teuthology.orchestra.run.smithi188.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T03:57:28.817 INFO:teuthology.orchestra.run.smithi188.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T03:57:28.831 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T03:57:28.835 INFO:teuthology.orchestra.run.smithi188.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T03:57:28.836 INFO:teuthology.orchestra.run.smithi188.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T03:57:28.838 INFO:teuthology.orchestra.run.smithi188.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T03:57:28.838 INFO:teuthology.orchestra.run.smithi188.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T03:57:28.838 INFO:teuthology.orchestra.run.smithi188.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T03:57:28.839 INFO:teuthology.orchestra.run.smithi188.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T03:57:28.839 INFO:teuthology.orchestra.run.smithi188.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T03:57:28.840 INFO:teuthology.orchestra.run.smithi188.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T03:57:28.849 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T03:57:28.851 INFO:teuthology.orchestra.run.smithi188.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T03:57:28.851 INFO:teuthology.orchestra.run.smithi188.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T03:57:28.855 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T03:57:28.859 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T03:57:28.864 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T03:57:28.868 INFO:teuthology.orchestra.run.smithi188.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T03:57:28.873 INFO:teuthology.orchestra.run.smithi188.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T03:57:28.873 INFO:teuthology.orchestra.run.smithi188.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T03:57:28.874 INFO:teuthology.orchestra.run.smithi188.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T03:57:28.874 INFO:teuthology.orchestra.run.smithi188.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T03:57:28.876 INFO:teuthology.orchestra.run.smithi188.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T03:57:28.877 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T03:57:28.877 INFO:teuthology.orchestra.run.smithi188.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T03:57:28.878 INFO:teuthology.orchestra.run.smithi188.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T03:57:28.880 INFO:teuthology.orchestra.run.smithi188.stdout:Get:46 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T03:57:28.881 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T03:57:28.882 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T03:57:28.883 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T03:57:28.885 INFO:teuthology.orchestra.run.smithi188.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T03:57:28.888 INFO:teuthology.orchestra.run.smithi188.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T03:57:28.891 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T03:57:28.892 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T03:57:28.894 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T03:57:28.896 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T03:57:28.897 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T03:57:28.898 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T03:57:28.898 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T03:57:28.900 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T03:57:28.901 INFO:teuthology.orchestra.run.smithi188.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T03:57:28.902 INFO:teuthology.orchestra.run.smithi188.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T03:57:28.903 INFO:teuthology.orchestra.run.smithi188.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T03:57:28.905 INFO:teuthology.orchestra.run.smithi188.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T03:57:28.909 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T03:57:28.914 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T03:57:28.915 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T03:57:28.915 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T03:57:28.915 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T03:57:28.917 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T03:57:28.917 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T03:57:28.918 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T03:57:28.919 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T03:57:28.923 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T03:57:28.926 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T03:57:28.928 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T03:57:28.930 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T03:57:28.930 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T03:57:28.932 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T03:57:28.933 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T03:57:28.933 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T03:57:28.933 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T03:57:28.933 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T03:57:28.934 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T03:57:28.934 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T03:57:28.947 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T03:57:28.949 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T03:57:28.950 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T03:57:28.950 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T03:57:28.950 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T03:57:28.952 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T03:57:28.952 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T03:57:28.953 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T03:57:28.953 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T03:57:28.953 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T03:57:28.953 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T03:57:28.960 INFO:teuthology.orchestra.run.smithi188.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T03:57:28.961 INFO:teuthology.orchestra.run.smithi188.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T03:57:28.962 INFO:teuthology.orchestra.run.smithi188.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T03:57:28.962 INFO:teuthology.orchestra.run.smithi188.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T03:57:28.963 INFO:teuthology.orchestra.run.smithi188.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T03:57:28.965 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T03:57:28.965 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T03:57:28.966 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T03:57:28.966 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T03:57:28.967 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T03:57:28.967 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T03:57:28.968 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T03:57:28.969 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T03:57:28.969 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T03:57:28.970 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T03:57:28.979 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T03:57:28.980 INFO:teuthology.orchestra.run.smithi188.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T03:57:28.980 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T03:57:28.982 INFO:teuthology.orchestra.run.smithi188.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T03:57:28.982 INFO:teuthology.orchestra.run.smithi188.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T03:57:28.982 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T03:57:28.984 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T03:57:28.984 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T03:57:28.985 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T03:57:28.985 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T03:57:28.986 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T03:57:28.986 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T03:57:28.986 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T03:57:28.987 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T03:57:28.989 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T03:57:28.989 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T03:57:28.989 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T03:57:28.990 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T03:57:28.993 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T03:57:28.996 INFO:teuthology.orchestra.run.smithi188.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T03:57:28.996 INFO:teuthology.orchestra.run.smithi188.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T03:57:28.999 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T03:57:29.000 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T03:57:29.001 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T03:57:29.001 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T03:57:29.002 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T03:57:29.002 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T03:57:29.003 INFO:teuthology.orchestra.run.smithi188.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T03:57:29.003 INFO:teuthology.orchestra.run.smithi188.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T03:57:29.003 INFO:teuthology.orchestra.run.smithi039.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T03:57:29.004 INFO:teuthology.orchestra.run.smithi188.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T03:57:29.004 INFO:teuthology.orchestra.run.smithi188.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T03:57:29.004 INFO:teuthology.orchestra.run.smithi188.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T03:57:29.005 INFO:teuthology.orchestra.run.smithi039.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T03:57:29.005 INFO:teuthology.orchestra.run.smithi188.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T03:57:29.005 INFO:teuthology.orchestra.run.smithi188.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T03:57:29.006 INFO:teuthology.orchestra.run.smithi039.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T03:57:29.008 INFO:teuthology.orchestra.run.smithi188.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T03:57:29.017 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T03:57:29.018 INFO:teuthology.orchestra.run.smithi039.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T03:57:29.018 INFO:teuthology.orchestra.run.smithi188.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T03:57:29.018 INFO:teuthology.orchestra.run.smithi039.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T03:57:29.019 INFO:teuthology.orchestra.run.smithi039.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T03:57:29.020 INFO:teuthology.orchestra.run.smithi188.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T03:57:29.022 INFO:teuthology.orchestra.run.smithi188.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T03:57:29.022 INFO:teuthology.orchestra.run.smithi188.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T03:57:29.022 INFO:teuthology.orchestra.run.smithi039.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T03:57:29.023 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T03:57:29.023 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T03:57:29.023 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T03:57:29.024 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T03:57:29.025 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T03:57:29.026 INFO:teuthology.orchestra.run.smithi188.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T03:57:29.027 INFO:teuthology.orchestra.run.smithi188.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T03:57:29.030 INFO:teuthology.orchestra.run.smithi188.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T03:57:29.032 INFO:teuthology.orchestra.run.smithi188.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T03:57:29.034 INFO:teuthology.orchestra.run.smithi037.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T03:57:29.035 INFO:teuthology.orchestra.run.smithi037.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T03:57:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T03:57:29.037 INFO:teuthology.orchestra.run.smithi188.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T03:57:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T03:57:29.039 INFO:teuthology.orchestra.run.smithi188.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T03:57:29.040 INFO:teuthology.orchestra.run.smithi188.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T03:57:29.040 INFO:teuthology.orchestra.run.smithi188.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T03:57:29.044 INFO:teuthology.orchestra.run.smithi188.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T03:57:29.045 INFO:teuthology.orchestra.run.smithi188.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T03:57:29.047 INFO:teuthology.orchestra.run.smithi188.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T03:57:29.051 INFO:teuthology.orchestra.run.smithi037.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T03:57:29.052 INFO:teuthology.orchestra.run.smithi037.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T03:57:29.053 INFO:teuthology.orchestra.run.smithi037.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T03:57:29.055 INFO:teuthology.orchestra.run.smithi188.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T03:57:29.056 INFO:teuthology.orchestra.run.smithi037.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T03:57:29.056 INFO:teuthology.orchestra.run.smithi188.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T03:57:29.056 INFO:teuthology.orchestra.run.smithi188.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T03:57:29.065 INFO:teuthology.orchestra.run.smithi188.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T03:57:29.073 INFO:teuthology.orchestra.run.smithi188.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T03:57:29.074 INFO:teuthology.orchestra.run.smithi188.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T03:57:29.074 INFO:teuthology.orchestra.run.smithi039.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T03:57:29.075 INFO:teuthology.orchestra.run.smithi039.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T03:57:29.076 INFO:teuthology.orchestra.run.smithi039.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T03:57:29.076 INFO:teuthology.orchestra.run.smithi039.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T03:57:29.078 INFO:teuthology.orchestra.run.smithi039.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T03:57:29.080 INFO:teuthology.orchestra.run.smithi188.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T03:57:29.080 INFO:teuthology.orchestra.run.smithi188.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T03:57:29.080 INFO:teuthology.orchestra.run.smithi188.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T03:57:29.082 INFO:teuthology.orchestra.run.smithi188.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T03:57:29.083 INFO:teuthology.orchestra.run.smithi188.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T03:57:29.095 INFO:teuthology.orchestra.run.smithi039.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T03:57:29.097 INFO:teuthology.orchestra.run.smithi039.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T03:57:29.097 INFO:teuthology.orchestra.run.smithi039.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T03:57:29.111 INFO:teuthology.orchestra.run.smithi037.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T03:57:29.111 INFO:teuthology.orchestra.run.smithi037.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T03:57:29.112 INFO:teuthology.orchestra.run.smithi039.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T03:57:29.112 INFO:teuthology.orchestra.run.smithi039.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T03:57:29.113 INFO:teuthology.orchestra.run.smithi039.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T03:57:29.114 INFO:teuthology.orchestra.run.smithi037.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T03:57:29.114 INFO:teuthology.orchestra.run.smithi037.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T03:57:29.115 INFO:teuthology.orchestra.run.smithi037.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T03:57:29.115 INFO:teuthology.orchestra.run.smithi039.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T03:57:29.115 INFO:teuthology.orchestra.run.smithi039.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T03:57:29.115 INFO:teuthology.orchestra.run.smithi039.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T03:57:29.122 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T03:57:29.127 INFO:teuthology.orchestra.run.smithi039.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T03:57:29.128 INFO:teuthology.orchestra.run.smithi039.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T03:57:29.131 INFO:teuthology.orchestra.run.smithi039.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T03:57:29.131 INFO:teuthology.orchestra.run.smithi037.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T03:57:29.133 INFO:teuthology.orchestra.run.smithi037.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T03:57:29.133 INFO:teuthology.orchestra.run.smithi037.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T03:57:29.135 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T03:57:29.144 INFO:teuthology.orchestra.run.smithi039.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T03:57:29.146 INFO:teuthology.orchestra.run.smithi039.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T03:57:29.146 INFO:teuthology.orchestra.run.smithi039.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T03:57:29.147 INFO:teuthology.orchestra.run.smithi037.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T03:57:29.148 INFO:teuthology.orchestra.run.smithi037.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T03:57:29.148 INFO:teuthology.orchestra.run.smithi037.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T03:57:29.149 INFO:teuthology.orchestra.run.smithi037.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T03:57:29.149 INFO:teuthology.orchestra.run.smithi037.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T03:57:29.149 INFO:teuthology.orchestra.run.smithi039.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T03:57:29.150 INFO:teuthology.orchestra.run.smithi037.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T03:57:29.151 INFO:teuthology.orchestra.run.smithi039.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T03:57:29.152 INFO:teuthology.orchestra.run.smithi039.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T03:57:29.152 INFO:teuthology.orchestra.run.smithi039.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T03:57:29.161 INFO:teuthology.orchestra.run.smithi039.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T03:57:29.162 INFO:teuthology.orchestra.run.smithi039.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T03:57:29.164 INFO:teuthology.orchestra.run.smithi037.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T03:57:29.165 INFO:teuthology.orchestra.run.smithi039.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T03:57:29.165 INFO:teuthology.orchestra.run.smithi037.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T03:57:29.168 INFO:teuthology.orchestra.run.smithi037.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T03:57:29.179 INFO:teuthology.orchestra.run.smithi039.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T03:57:29.180 INFO:teuthology.orchestra.run.smithi039.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T03:57:29.180 INFO:teuthology.orchestra.run.smithi039.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T03:57:29.181 INFO:teuthology.orchestra.run.smithi039.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T03:57:29.181 INFO:teuthology.orchestra.run.smithi037.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T03:57:29.182 INFO:teuthology.orchestra.run.smithi039.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T03:57:29.183 INFO:teuthology.orchestra.run.smithi037.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T03:57:29.183 INFO:teuthology.orchestra.run.smithi037.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T03:57:29.186 INFO:teuthology.orchestra.run.smithi037.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T03:57:29.187 INFO:teuthology.orchestra.run.smithi039.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T03:57:29.188 INFO:teuthology.orchestra.run.smithi039.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T03:57:29.188 INFO:teuthology.orchestra.run.smithi037.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T03:57:29.189 INFO:teuthology.orchestra.run.smithi037.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T03:57:29.189 INFO:teuthology.orchestra.run.smithi037.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T03:57:29.195 INFO:teuthology.orchestra.run.smithi039.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T03:57:29.197 INFO:teuthology.orchestra.run.smithi039.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T03:57:29.198 INFO:teuthology.orchestra.run.smithi039.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T03:57:29.199 INFO:teuthology.orchestra.run.smithi037.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T03:57:29.199 INFO:teuthology.orchestra.run.smithi037.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T03:57:29.202 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T03:57:29.202 INFO:teuthology.orchestra.run.smithi037.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T03:57:29.218 INFO:teuthology.orchestra.run.smithi117.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T03:57:29.218 INFO:teuthology.orchestra.run.smithi037.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T03:57:29.218 INFO:teuthology.orchestra.run.smithi037.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T03:57:29.218 INFO:teuthology.orchestra.run.smithi037.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T03:57:29.219 INFO:teuthology.orchestra.run.smithi037.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T03:57:29.219 INFO:teuthology.orchestra.run.smithi037.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T03:57:29.225 INFO:teuthology.orchestra.run.smithi037.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T03:57:29.225 INFO:teuthology.orchestra.run.smithi037.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T03:57:29.230 INFO:teuthology.orchestra.run.smithi117.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T03:57:29.235 INFO:teuthology.orchestra.run.smithi037.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T03:57:29.237 INFO:teuthology.orchestra.run.smithi037.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T03:57:29.237 INFO:teuthology.orchestra.run.smithi037.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T03:57:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T03:57:29.304 INFO:teuthology.orchestra.run.smithi188.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T03:57:29.307 INFO:teuthology.orchestra.run.smithi188.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T03:57:29.316 INFO:teuthology.orchestra.run.smithi188.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T03:57:29.445 INFO:teuthology.orchestra.run.smithi117.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T03:57:29.455 INFO:teuthology.orchestra.run.smithi117.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T03:57:29.485 INFO:teuthology.orchestra.run.smithi117.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T03:57:29.494 INFO:teuthology.orchestra.run.smithi117.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T03:57:29.495 INFO:teuthology.orchestra.run.smithi117.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T03:57:29.496 INFO:teuthology.orchestra.run.smithi117.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T03:57:29.514 INFO:teuthology.orchestra.run.smithi117.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T03:57:29.518 INFO:teuthology.orchestra.run.smithi117.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T03:57:29.522 INFO:teuthology.orchestra.run.smithi117.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T03:57:29.526 INFO:teuthology.orchestra.run.smithi117.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T03:57:29.607 INFO:teuthology.orchestra.run.smithi117.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T03:57:29.611 INFO:teuthology.orchestra.run.smithi117.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T03:57:29.615 INFO:teuthology.orchestra.run.smithi117.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T03:57:29.619 INFO:teuthology.orchestra.run.smithi117.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T03:57:29.620 INFO:teuthology.orchestra.run.smithi117.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T03:57:29.621 INFO:teuthology.orchestra.run.smithi117.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T03:57:29.622 INFO:teuthology.orchestra.run.smithi117.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T03:57:29.623 INFO:teuthology.orchestra.run.smithi117.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T03:57:29.623 INFO:teuthology.orchestra.run.smithi117.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T03:57:29.690 INFO:teuthology.orchestra.run.smithi117.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T03:57:29.711 INFO:teuthology.orchestra.run.smithi117.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T03:57:29.712 INFO:teuthology.orchestra.run.smithi117.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T03:57:29.712 INFO:teuthology.orchestra.run.smithi117.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T03:57:29.713 INFO:teuthology.orchestra.run.smithi117.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T03:57:29.714 INFO:teuthology.orchestra.run.smithi117.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T03:57:29.715 INFO:teuthology.orchestra.run.smithi117.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T03:57:29.772 INFO:teuthology.orchestra.run.smithi117.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T03:57:29.773 INFO:teuthology.orchestra.run.smithi117.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T03:57:29.774 INFO:teuthology.orchestra.run.smithi117.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T03:57:29.853 INFO:teuthology.orchestra.run.smithi117.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T03:57:29.855 INFO:teuthology.orchestra.run.smithi117.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T03:57:29.856 INFO:teuthology.orchestra.run.smithi117.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T03:57:29.857 INFO:teuthology.orchestra.run.smithi117.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T03:57:29.868 INFO:teuthology.orchestra.run.smithi117.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T03:57:29.868 INFO:teuthology.orchestra.run.smithi117.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T03:57:29.934 INFO:teuthology.orchestra.run.smithi117.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T03:57:29.934 INFO:teuthology.orchestra.run.smithi117.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T03:57:29.937 INFO:teuthology.orchestra.run.smithi117.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T03:57:29.941 INFO:teuthology.orchestra.run.smithi117.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T03:57:30.015 INFO:teuthology.orchestra.run.smithi117.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T03:57:30.018 INFO:teuthology.orchestra.run.smithi117.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T03:57:30.019 INFO:teuthology.orchestra.run.smithi117.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T03:57:30.023 INFO:teuthology.orchestra.run.smithi117.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T03:57:30.025 INFO:teuthology.orchestra.run.smithi117.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T03:57:30.026 INFO:teuthology.orchestra.run.smithi117.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T03:57:30.096 INFO:teuthology.orchestra.run.smithi117.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T03:57:30.214 INFO:teuthology.orchestra.run.smithi117.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T03:57:30.231 INFO:teuthology.orchestra.run.smithi188.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T03:57:30.295 INFO:teuthology.orchestra.run.smithi117.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T03:57:30.311 INFO:teuthology.orchestra.run.smithi188.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T03:57:30.313 INFO:teuthology.orchestra.run.smithi188.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T03:57:30.316 INFO:teuthology.orchestra.run.smithi188.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T03:57:30.357 INFO:teuthology.orchestra.run.smithi188.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T03:57:30.358 INFO:teuthology.orchestra.run.smithi117.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T03:57:30.359 INFO:teuthology.orchestra.run.smithi117.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T03:57:30.359 INFO:teuthology.orchestra.run.smithi117.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T03:57:30.361 INFO:teuthology.orchestra.run.smithi117.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T03:57:30.408 INFO:teuthology.orchestra.run.smithi117.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T03:57:30.412 INFO:teuthology.orchestra.run.smithi117.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T03:57:30.412 INFO:teuthology.orchestra.run.smithi117.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T03:57:30.457 INFO:teuthology.orchestra.run.smithi117.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T03:57:30.457 INFO:teuthology.orchestra.run.smithi117.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T03:57:30.458 INFO:teuthology.orchestra.run.smithi117.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T03:57:30.461 INFO:teuthology.orchestra.run.smithi117.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T03:57:30.469 INFO:teuthology.orchestra.run.smithi117.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T03:57:30.471 INFO:teuthology.orchestra.run.smithi117.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T03:57:30.521 INFO:teuthology.orchestra.run.smithi117.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T03:57:30.522 INFO:teuthology.orchestra.run.smithi117.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T03:57:30.523 INFO:teuthology.orchestra.run.smithi117.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T03:57:30.523 INFO:teuthology.orchestra.run.smithi117.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T03:57:30.525 INFO:teuthology.orchestra.run.smithi117.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T03:57:30.531 INFO:teuthology.orchestra.run.smithi117.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T03:57:30.539 INFO:teuthology.orchestra.run.smithi188.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T03:57:30.546 INFO:teuthology.orchestra.run.smithi117.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T03:57:30.603 INFO:teuthology.orchestra.run.smithi117.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T03:57:30.606 INFO:teuthology.orchestra.run.smithi117.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T03:57:30.607 INFO:teuthology.orchestra.run.smithi117.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T03:57:30.616 INFO:teuthology.orchestra.run.smithi117.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T03:57:30.684 INFO:teuthology.orchestra.run.smithi117.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T03:57:30.688 INFO:teuthology.orchestra.run.smithi117.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T03:57:30.689 INFO:teuthology.orchestra.run.smithi117.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T03:57:30.701 INFO:teuthology.orchestra.run.smithi117.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T03:57:30.702 INFO:teuthology.orchestra.run.smithi117.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T03:57:30.709 INFO:teuthology.orchestra.run.smithi117.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T03:57:30.737 INFO:teuthology.orchestra.run.smithi039.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T03:57:30.768 INFO:teuthology.orchestra.run.smithi117.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T03:57:30.769 INFO:teuthology.orchestra.run.smithi117.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T03:57:30.770 INFO:teuthology.orchestra.run.smithi117.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T03:57:30.774 INFO:teuthology.orchestra.run.smithi117.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T03:57:30.780 INFO:teuthology.orchestra.run.smithi117.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T03:57:30.849 INFO:teuthology.orchestra.run.smithi117.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T03:57:30.866 INFO:teuthology.orchestra.run.smithi117.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T03:57:30.866 INFO:teuthology.orchestra.run.smithi117.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T03:57:30.867 INFO:teuthology.orchestra.run.smithi117.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T03:57:30.872 INFO:teuthology.orchestra.run.smithi117.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T03:57:30.874 INFO:teuthology.orchestra.run.smithi117.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T03:57:31.103 INFO:teuthology.orchestra.run.smithi037.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T03:57:31.363 INFO:teuthology.orchestra.run.smithi188.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T03:57:31.364 INFO:teuthology.orchestra.run.smithi188.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T03:57:31.381 INFO:teuthology.orchestra.run.smithi188.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T03:57:31.424 INFO:teuthology.orchestra.run.smithi188.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T03:57:31.437 INFO:teuthology.orchestra.run.smithi188.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T03:57:31.439 INFO:teuthology.orchestra.run.smithi188.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T03:57:31.513 INFO:teuthology.orchestra.run.smithi188.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi117.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi117.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi117.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T03:57:31.765 INFO:teuthology.orchestra.run.smithi117.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T03:57:31.772 INFO:teuthology.orchestra.run.smithi117.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T03:57:31.774 INFO:teuthology.orchestra.run.smithi117.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T03:57:31.783 INFO:teuthology.orchestra.run.smithi188.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T03:57:31.784 INFO:teuthology.orchestra.run.smithi188.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T03:57:31.881 INFO:teuthology.orchestra.run.smithi117.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T03:57:32.175 INFO:teuthology.orchestra.run.smithi117.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T03:57:32.176 INFO:teuthology.orchestra.run.smithi117.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T03:57:32.381 INFO:teuthology.orchestra.run.smithi039.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T03:57:32.730 INFO:teuthology.orchestra.run.smithi039.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T03:57:32.942 INFO:teuthology.orchestra.run.smithi039.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T03:57:32.943 INFO:teuthology.orchestra.run.smithi039.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T03:57:33.014 INFO:teuthology.orchestra.run.smithi037.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T03:57:33.084 INFO:teuthology.orchestra.run.smithi039.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T03:57:33.086 INFO:teuthology.orchestra.run.smithi039.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T03:57:33.302 INFO:teuthology.orchestra.run.smithi039.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T03:57:33.450 INFO:teuthology.orchestra.run.smithi037.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T03:57:33.654 INFO:teuthology.orchestra.run.smithi037.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T03:57:33.656 INFO:teuthology.orchestra.run.smithi037.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T03:57:33.709 INFO:teuthology.orchestra.run.smithi037.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T03:57:33.740 INFO:teuthology.orchestra.run.smithi037.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T03:57:33.855 INFO:teuthology.orchestra.run.smithi037.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T03:57:33.958 INFO:teuthology.orchestra.run.smithi188.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T03:57:34.034 INFO:teuthology.orchestra.run.smithi188.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T03:57:34.035 INFO:teuthology.orchestra.run.smithi188.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T03:57:34.093 INFO:teuthology.orchestra.run.smithi117.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T03:57:34.095 INFO:teuthology.orchestra.run.smithi117.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T03:57:34.096 INFO:teuthology.orchestra.run.smithi117.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T03:57:34.333 INFO:teuthology.orchestra.run.smithi188.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T03:57:34.412 INFO:teuthology.orchestra.run.smithi117.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T03:57:34.784 INFO:teuthology.orchestra.run.smithi039.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T03:57:34.790 INFO:teuthology.orchestra.run.smithi039.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T03:57:34.852 INFO:teuthology.orchestra.run.smithi039.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T03:57:34.901 INFO:teuthology.orchestra.run.smithi188.stdout:Fetched 181 MB in 6s (31.1 MB/s) 2024-09-16T03:57:34.981 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 181 MB in 6s (30.7 MB/s) 2024-09-16T03:57:35.015 INFO:teuthology.orchestra.run.smithi037.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T03:57:35.018 INFO:teuthology.orchestra.run.smithi037.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T03:57:35.075 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T03:57:35.080 INFO:teuthology.orchestra.run.smithi037.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T03:57:35.139 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T03:57:35.176 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T03:57:35.182 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T03:57:35.239 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T03:57:35.244 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T03:57:35.273 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:57:35.329 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:57:35.552 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T03:57:35.578 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T03:57:35.610 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:57:35.666 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T03:57:35.692 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T03:57:35.733 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:57:35.938 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T03:57:35.965 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T03:57:36.005 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:57:36.044 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T03:57:36.070 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T03:57:36.119 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:57:36.442 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T03:57:36.468 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:36.516 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:36.531 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T03:57:36.557 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:36.606 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:36.895 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T03:57:36.921 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:36.961 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:37.009 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T03:57:37.035 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:37.075 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:37.353 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T03:57:37.379 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:37.411 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T03:57:37.598 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T03:57:37.625 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:37.665 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:37.704 INFO:teuthology.orchestra.run.smithi037.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T03:57:37.739 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T03:57:37.766 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T03:57:37.806 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:57:37.837 INFO:teuthology.orchestra.run.smithi039.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T03:57:37.842 INFO:teuthology.orchestra.run.smithi039.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T03:57:37.860 INFO:teuthology.orchestra.run.smithi039.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T03:57:37.983 INFO:teuthology.orchestra.run.smithi039.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T03:57:38.017 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T03:57:38.044 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T03:57:38.084 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:57:38.088 INFO:teuthology.orchestra.run.smithi037.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T03:57:38.108 INFO:teuthology.orchestra.run.smithi037.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T03:57:38.111 INFO:teuthology.orchestra.run.smithi037.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T03:57:38.134 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librados2. 2024-09-16T03:57:38.160 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:38.200 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:38.234 INFO:teuthology.orchestra.run.smithi037.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T03:57:38.378 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package librados2. 2024-09-16T03:57:38.394 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:38.422 INFO:teuthology.orchestra.run.smithi039.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T03:57:38.437 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:38.645 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librbd1. 2024-09-16T03:57:38.671 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:38.686 INFO:teuthology.orchestra.run.smithi037.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T03:57:38.711 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:38.857 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package librbd1. 2024-09-16T03:57:38.883 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:38.923 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:39.147 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T03:57:39.174 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:39.214 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:39.368 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T03:57:39.394 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:39.434 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:39.491 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rados. 2024-09-16T03:57:39.506 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:39.541 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:39.712 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-rados. 2024-09-16T03:57:39.727 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:39.770 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:39.836 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T03:57:39.863 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:39.902 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:39.993 INFO:teuthology.orchestra.run.smithi039.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T03:57:39.994 INFO:teuthology.orchestra.run.smithi039.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T03:57:40.018 INFO:teuthology.orchestra.run.smithi039.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T03:57:40.048 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T03:57:40.075 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:40.106 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:40.163 INFO:teuthology.orchestra.run.smithi039.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T03:57:40.186 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T03:57:40.205 INFO:teuthology.orchestra.run.smithi039.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T03:57:40.213 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:40.222 INFO:teuthology.orchestra.run.smithi039.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T03:57:40.255 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:40.285 INFO:teuthology.orchestra.run.smithi037.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T03:57:40.286 INFO:teuthology.orchestra.run.smithi037.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T03:57:40.308 INFO:teuthology.orchestra.run.smithi037.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T03:57:40.350 INFO:teuthology.orchestra.run.smithi039.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T03:57:40.350 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T03:57:40.365 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:40.409 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:40.453 INFO:teuthology.orchestra.run.smithi037.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T03:57:40.497 INFO:teuthology.orchestra.run.smithi037.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T03:57:40.511 INFO:teuthology.orchestra.run.smithi037.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T03:57:40.566 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T03:57:40.592 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:40.632 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:40.641 INFO:teuthology.orchestra.run.smithi037.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T03:57:40.728 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T03:57:40.755 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:40.795 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:40.843 INFO:teuthology.orchestra.run.smithi039.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T03:57:40.843 INFO:teuthology.orchestra.run.smithi039.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T03:57:40.952 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T03:57:40.979 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T03:57:41.019 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:57:41.115 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T03:57:41.140 INFO:teuthology.orchestra.run.smithi037.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T03:57:41.140 INFO:teuthology.orchestra.run.smithi037.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T03:57:41.141 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T03:57:41.181 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:57:41.296 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T03:57:41.323 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T03:57:41.354 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T03:57:41.450 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T03:57:41.466 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T03:57:41.500 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T03:57:41.599 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T03:57:41.625 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:41.665 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:41.770 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T03:57:41.797 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:41.837 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:42.001 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T03:57:42.028 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T03:57:42.068 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:57:42.173 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T03:57:42.200 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T03:57:42.240 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:57:42.534 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T03:57:42.561 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T03:57:42.595 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T03:57:42.601 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:57:42.611 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T03:57:42.654 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:57:42.862 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T03:57:42.877 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T03:57:42.921 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:57:42.923 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T03:57:42.938 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T03:57:42.973 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:57:43.249 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package lua5.1. 2024-09-16T03:57:43.276 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T03:57:43.293 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package lua5.1. 2024-09-16T03:57:43.319 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T03:57:43.324 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:57:43.360 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:57:43.507 INFO:teuthology.orchestra.run.smithi039.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T03:57:43.510 INFO:teuthology.orchestra.run.smithi039.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T03:57:43.511 INFO:teuthology.orchestra.run.smithi039.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T03:57:43.644 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package lua-any. 2024-09-16T03:57:43.651 INFO:teuthology.orchestra.run.smithi037.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T03:57:43.653 INFO:teuthology.orchestra.run.smithi037.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T03:57:43.655 INFO:teuthology.orchestra.run.smithi037.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T03:57:43.659 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T03:57:43.671 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package lua-any. 2024-09-16T03:57:43.686 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T03:57:43.702 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T03:57:43.746 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T03:57:43.901 INFO:teuthology.orchestra.run.smithi039.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T03:57:43.940 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package zip. 2024-09-16T03:57:43.948 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package zip. 2024-09-16T03:57:43.955 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T03:57:43.964 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T03:57:43.998 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T03:57:43.998 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T03:57:43.999 INFO:teuthology.orchestra.run.smithi037.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T03:57:44.309 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package unzip. 2024-09-16T03:57:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package unzip. 2024-09-16T03:57:44.336 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T03:57:44.361 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T03:57:44.376 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:57:44.401 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:57:44.505 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 181 MB in 15s (11.9 MB/s) 2024-09-16T03:57:44.594 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 181 MB in 15s (11.8 MB/s) 2024-09-16T03:57:44.642 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T03:57:44.745 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T03:57:44.747 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package luarocks. 2024-09-16T03:57:44.750 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T03:57:44.763 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package luarocks. 2024-09-16T03:57:44.773 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T03:57:44.790 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T03:57:44.813 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:57:44.830 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:57:44.832 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:57:44.856 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T03:57:44.960 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T03:57:44.965 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T03:57:45.037 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:57:45.154 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package librgw2. 2024-09-16T03:57:45.181 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:45.186 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T03:57:45.196 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librgw2. 2024-09-16T03:57:45.213 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T03:57:45.223 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:45.246 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:45.253 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:57:45.262 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:45.374 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T03:57:45.401 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T03:57:45.441 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:57:45.631 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T03:57:45.658 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T03:57:45.697 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:57:45.752 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T03:57:45.779 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T03:57:45.790 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T03:57:45.806 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:45.823 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T03:57:45.839 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:45.844 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:57:45.857 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:45.873 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:46.101 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T03:57:46.128 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:46.168 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T03:57:46.176 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:46.193 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T03:57:46.194 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T03:57:46.197 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T03:57:46.220 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T03:57:46.224 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:46.234 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:57:46.260 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:57:46.272 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:46.521 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T03:57:46.536 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:46.545 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T03:57:46.570 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T03:57:46.571 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:46.571 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:46.598 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:46.612 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:46.619 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T03:57:46.637 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:46.644 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:46.675 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:46.965 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T03:57:46.991 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-common. 2024-09-16T03:57:46.991 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:46.995 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T03:57:47.018 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:47.022 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:57:47.031 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:47.040 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-common. 2024-09-16T03:57:47.057 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:47.061 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:57:47.067 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:47.107 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:47.360 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T03:57:47.386 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T03:57:47.426 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:57:47.515 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T03:57:47.542 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T03:57:47.581 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:57:47.746 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librados2. 2024-09-16T03:57:48.027 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:48.027 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:48.027 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librados2. 2024-09-16T03:57:48.027 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:48.028 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:48.281 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librbd1. 2024-09-16T03:57:48.298 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:48.329 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librbd1. 2024-09-16T03:57:48.340 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:48.355 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:48.395 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:48.438 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-base. 2024-09-16T03:57:48.445 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-base. 2024-09-16T03:57:48.464 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:48.472 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:48.514 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:48.521 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:48.743 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T03:57:48.769 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:48.801 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T03:57:48.849 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:48.889 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:49.045 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rados. 2024-09-16T03:57:49.060 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:49.067 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T03:57:49.074 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T03:57:49.093 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T03:57:49.101 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T03:57:49.103 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:49.126 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:57:49.141 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:57:49.176 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rados. 2024-09-16T03:57:49.192 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:49.225 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:49.395 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T03:57:49.421 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T03:57:49.444 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T03:57:49.461 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:57:49.471 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T03:57:49.487 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T03:57:49.510 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:57:49.513 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:49.554 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:49.572 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T03:57:49.599 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:49.639 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:49.765 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T03:57:49.791 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T03:57:49.822 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T03:57:49.830 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T03:57:49.831 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:57:49.838 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:49.857 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T03:57:49.880 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:49.891 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T03:57:49.896 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:57:49.918 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:49.958 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:50.141 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T03:57:50.142 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T03:57:50.158 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T03:57:50.167 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T03:57:50.170 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T03:57:50.185 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:50.199 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:57:50.201 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:57:50.225 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:50.236 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T03:57:50.252 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:50.287 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:50.502 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T03:57:50.503 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T03:57:50.512 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T03:57:50.519 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T03:57:50.529 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T03:57:50.539 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T03:57:50.553 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:57:50.569 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:57:50.572 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T03:57:50.578 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:57:50.600 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T03:57:50.639 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:57:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T03:57:50.871 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T03:57:50.881 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T03:57:50.882 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T03:57:50.898 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T03:57:50.908 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T03:57:50.931 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T03:57:50.938 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T03:57:50.948 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T03:57:50.975 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T03:57:51.001 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T03:57:51.041 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T03:57:51.200 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T03:57:51.216 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-portend. 2024-09-16T03:57:51.226 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-portend. 2024-09-16T03:57:51.227 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:51.231 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T03:57:51.253 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T03:57:51.265 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T03:57:51.266 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:51.293 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T03:57:51.302 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T03:57:51.318 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:51.361 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:51.527 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T03:57:51.554 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T03:57:51.570 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T03:57:51.593 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T03:57:51.597 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T03:57:51.611 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T03:57:51.637 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T03:57:51.637 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T03:57:51.678 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:57:51.680 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T03:57:51.707 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T03:57:51.738 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:57:51.896 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T03:57:51.914 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T03:57:51.923 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T03:57:51.929 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T03:57:51.940 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T03:57:51.963 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:57:51.964 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:57:51.967 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T03:57:51.999 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T03:57:52.006 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:57:52.026 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T03:57:52.065 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:57:52.250 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T03:57:52.266 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T03:57:52.266 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T03:57:52.282 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T03:57:52.308 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:57:52.310 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T03:57:52.316 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T03:57:52.326 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T03:57:52.352 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T03:57:52.368 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T03:57:52.369 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T03:57:52.402 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:57:52.653 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T03:57:52.679 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T03:57:52.680 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T03:57:52.706 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T03:57:52.722 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua5.1. 2024-09-16T03:57:52.746 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T03:57:52.748 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T03:57:52.786 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua5.1. 2024-09-16T03:57:52.788 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:57:52.813 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T03:57:52.832 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T03:57:52.853 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:57:53.033 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T03:57:53.060 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T03:57:53.099 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:57:53.107 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-any. 2024-09-16T03:57:53.123 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T03:57:53.163 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T03:57:53.164 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-any. 2024-09-16T03:57:53.166 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T03:57:53.190 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T03:57:53.191 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T03:57:53.230 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:57:53.231 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T03:57:53.377 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-mako. 2024-09-16T03:57:53.404 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T03:57:53.419 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package zip. 2024-09-16T03:57:53.435 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T03:57:53.443 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:57:53.450 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package zip. 2024-09-16T03:57:53.466 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T03:57:53.477 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T03:57:53.533 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-mako. 2024-09-16T03:57:53.534 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T03:57:53.560 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T03:57:53.599 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:57:53.755 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T03:57:53.782 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T03:57:53.792 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package unzip. 2024-09-16T03:57:53.813 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:57:53.815 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T03:57:53.845 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package unzip. 2024-09-16T03:57:53.855 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:57:53.872 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T03:57:53.912 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:57:53.928 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T03:57:53.955 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T03:57:53.994 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:57:54.116 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T03:57:54.143 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T03:57:54.183 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:57:54.242 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package luarocks. 2024-09-16T03:57:54.269 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T03:57:54.272 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T03:57:54.287 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T03:57:54.299 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package luarocks. 2024-09-16T03:57:54.308 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:57:54.322 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:57:54.325 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T03:57:54.365 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:57:54.486 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-webob. 2024-09-16T03:57:54.513 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T03:57:54.552 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:57:54.558 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webob. 2024-09-16T03:57:54.585 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T03:57:54.616 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:57:54.666 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librgw2. 2024-09-16T03:57:54.693 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:54.733 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:54.748 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librgw2. 2024-09-16T03:57:54.775 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:54.815 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:54.897 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T03:57:54.924 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T03:57:54.936 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T03:57:54.963 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T03:57:54.967 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:57:55.005 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:57:55.277 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T03:57:55.283 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T03:57:55.293 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:55.305 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T03:57:55.310 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T03:57:55.332 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T03:57:55.335 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:55.343 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T03:57:55.350 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:57:55.369 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:55.371 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:57:55.401 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:55.620 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-paste. 2024-09-16T03:57:55.646 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T03:57:55.674 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-paste. 2024-09-16T03:57:55.678 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:57:55.688 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T03:57:55.701 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T03:57:55.712 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T03:57:55.715 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T03:57:55.739 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T03:57:55.741 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:57:55.755 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:57:55.778 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:57:56.006 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T03:57:56.032 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T03:57:56.033 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T03:57:56.059 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:56.073 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:57:56.077 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T03:57:56.091 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:56.098 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T03:57:56.105 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T03:57:56.125 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:56.144 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:57:56.165 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:56.376 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T03:57:56.402 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T03:57:56.405 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T03:57:56.431 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T03:57:56.442 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:57:56.472 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:57:56.510 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-common. 2024-09-16T03:57:56.533 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:56.552 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-common. 2024-09-16T03:57:56.568 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:56.579 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:56.619 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:56.729 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T03:57:56.750 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T03:57:56.755 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T03:57:56.765 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T03:57:56.795 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T03:57:56.843 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T03:57:57.082 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T03:57:57.109 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T03:57:57.140 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:57:57.153 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T03:57:57.180 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T03:57:57.219 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:57:57.444 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T03:57:57.460 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T03:57:57.494 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T03:57:57.557 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T03:57:57.584 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T03:57:57.624 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T03:57:57.814 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-idna. 2024-09-16T03:57:57.840 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T03:57:57.880 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:57:57.943 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-idna. 2024-09-16T03:57:57.970 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T03:57:58.008 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-base. 2024-09-16T03:57:58.019 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:57:58.034 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:58.091 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:58.158 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-base. 2024-09-16T03:57:58.184 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:58.235 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T03:57:58.238 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:58.257 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T03:57:58.292 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:57:58.330 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T03:57:58.357 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T03:57:58.406 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:57:58.670 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T03:57:58.697 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T03:57:58.724 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-requests. 2024-09-16T03:57:58.737 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:57:58.751 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T03:57:58.790 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:57:58.812 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-requests. 2024-09-16T03:57:58.838 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T03:57:58.878 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:57:59.014 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T03:57:59.041 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T03:57:59.072 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:57:59.077 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T03:57:59.104 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T03:57:59.135 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:57:59.148 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T03:57:59.174 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T03:57:59.215 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:57:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T03:57:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T03:57:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:57:59.430 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T03:57:59.445 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:59.486 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T03:57:59.488 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:59.513 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T03:57:59.526 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T03:57:59.542 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:57:59.552 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:57:59.584 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:59.813 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T03:57:59.818 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T03:57:59.829 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T03:57:59.844 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T03:57:59.863 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:57:59.909 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T03:57:59.937 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:57:59.937 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:57:59.973 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T03:57:59.975 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:57:59.999 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:00.048 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:00.125 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T03:58:00.152 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T03:58:00.200 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:58:00.240 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T03:58:00.266 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T03:58:00.287 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T03:58:00.306 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:58:00.314 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T03:58:00.353 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:58:00.359 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T03:58:00.386 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T03:58:00.425 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:58:00.503 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T03:58:00.529 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T03:58:00.569 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:58:00.584 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T03:58:00.610 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T03:58:00.642 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T03:58:00.682 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T03:58:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T03:58:00.708 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T03:58:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T03:58:00.748 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T03:58:00.770 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T03:58:00.872 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T03:58:00.899 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T03:58:00.939 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:58:00.939 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-portend. 2024-09-16T03:58:00.966 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T03:58:01.005 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T03:58:01.036 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T03:58:01.050 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:01.081 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T03:58:01.084 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:01.108 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:01.156 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:01.225 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T03:58:01.252 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T03:58:01.291 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T03:58:01.292 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T03:58:01.318 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T03:58:01.358 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T03:58:01.462 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T03:58:01.490 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:01.529 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:01.568 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T03:58:01.594 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-portend. 2024-09-16T03:58:01.595 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:01.621 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T03:58:01.634 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:01.661 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T03:58:01.686 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T03:58:01.713 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T03:58:01.753 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:58:01.964 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T03:58:01.991 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T03:58:02.030 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T03:58:02.123 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T03:58:02.150 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T03:58:02.190 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T03:58:02.213 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T03:58:02.240 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:02.279 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:02.309 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T03:58:02.335 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T03:58:02.367 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:58:02.501 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T03:58:02.528 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T03:58:02.568 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T03:58:02.712 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T03:58:02.739 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T03:58:02.778 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T03:58:02.854 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T03:58:02.881 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T03:58:02.908 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T03:58:02.912 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:58:02.935 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:03.090 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T03:58:03.116 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T03:58:03.156 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T03:58:03.182 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:03.190 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-mako. 2024-09-16T03:58:03.217 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T03:58:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:58:03.265 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph. 2024-09-16T03:58:03.281 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:03.324 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:03.418 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T03:58:03.444 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T03:58:03.476 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:58:03.585 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T03:58:03.612 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T03:58:03.643 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T03:58:03.652 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:58:03.671 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:03.710 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:03.778 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-mako. 2024-09-16T03:58:03.805 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T03:58:03.845 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:58:03.963 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T03:58:03.990 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T03:58:04.030 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:58:04.080 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T03:58:04.109 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:04.146 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:04.173 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T03:58:04.201 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T03:58:04.238 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph. 2024-09-16T03:58:04.240 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:58:04.253 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:04.266 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webob. 2024-09-16T03:58:04.282 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T03:58:04.287 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:04.316 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:58:04.483 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package cephadm. 2024-09-16T03:58:04.498 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:04.541 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:04.543 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T03:58:04.570 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T03:58:04.598 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T03:58:04.610 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:58:04.625 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:04.627 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T03:58:04.654 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T03:58:04.665 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:04.698 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:58:04.852 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T03:58:04.862 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webob. 2024-09-16T03:58:04.879 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T03:58:04.889 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T03:58:04.918 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T03:58:04.920 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:58:05.039 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T03:58:05.066 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T03:58:05.076 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T03:58:05.103 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:05.114 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:58:05.142 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:05.213 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T03:58:05.232 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T03:58:05.240 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:58:05.259 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T03:58:05.280 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:05.302 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:58:05.417 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-paste. 2024-09-16T03:58:05.444 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T03:58:05.483 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:58:05.546 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package cephadm. 2024-09-16T03:58:05.572 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:05.610 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T03:58:05.612 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:05.637 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T03:58:05.677 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:58:05.795 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T03:58:05.822 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T03:58:05.856 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T03:58:05.861 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:58:05.871 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T03:58:05.914 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T03:58:05.963 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-paste. 2024-09-16T03:58:05.990 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T03:58:06.029 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:58:06.140 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T03:58:06.167 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T03:58:06.206 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:58:06.225 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T03:58:06.252 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:58:06.292 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:06.324 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T03:58:06.335 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-babel. 2024-09-16T03:58:06.339 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T03:58:06.362 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:58:06.382 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:58:06.401 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:06.493 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T03:58:06.520 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T03:58:06.559 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T03:58:06.677 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T03:58:06.704 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T03:58:06.704 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T03:58:06.732 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T03:58:06.743 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:58:06.770 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:58:06.871 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T03:58:06.897 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T03:58:06.946 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:58:07.046 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T03:58:07.073 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T03:58:07.113 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T03:58:07.115 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T03:58:07.143 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:07.182 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:07.280 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-babel. 2024-09-16T03:58:07.301 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T03:58:07.307 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:58:07.328 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T03:58:07.346 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:07.367 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T03:58:07.382 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T03:58:07.398 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T03:58:07.432 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:58:07.519 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T03:58:07.546 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T03:58:07.585 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T03:58:07.649 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T03:58:07.677 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T03:58:07.695 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-idna. 2024-09-16T03:58:07.716 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:58:07.722 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T03:58:07.745 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T03:58:07.762 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:58:07.773 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T03:58:07.803 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T03:58:07.888 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-routes. 2024-09-16T03:58:07.915 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T03:58:07.955 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:58:08.073 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T03:58:08.077 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T03:58:08.100 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T03:58:08.104 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:08.115 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-idna. 2024-09-16T03:58:08.140 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:58:08.142 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T03:58:08.143 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:08.181 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:58:08.291 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T03:58:08.319 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:08.358 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:08.443 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests. 2024-09-16T03:58:08.470 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T03:58:08.480 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T03:58:08.492 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T03:58:08.507 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T03:58:08.509 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:58:08.519 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T03:58:08.547 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T03:58:08.567 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:58:08.774 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-routes. 2024-09-16T03:58:08.790 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T03:58:08.824 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:58:08.829 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T03:58:08.856 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T03:58:08.896 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:58:08.912 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests. 2024-09-16T03:58:08.939 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T03:58:08.979 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:58:09.152 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T03:58:09.179 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:09.182 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T03:58:09.198 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:09.210 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:09.241 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:09.290 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T03:58:09.317 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T03:58:09.357 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:58:09.621 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T03:58:09.648 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:09.651 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T03:58:09.667 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:09.696 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:09.710 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:09.796 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T03:58:09.812 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T03:58:09.846 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:58:10.007 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T03:58:10.034 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T03:58:10.073 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:58:10.123 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T03:58:10.151 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:10.189 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:10.325 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T03:58:10.352 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T03:58:10.402 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T03:58:10.416 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:58:10.429 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T03:58:10.432 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T03:58:10.459 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T03:58:10.485 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T03:58:10.498 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:58:10.500 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T03:58:10.528 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T03:58:10.567 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:58:10.778 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T03:58:10.796 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T03:58:10.805 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T03:58:10.823 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:10.844 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:58:10.871 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:10.879 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T03:58:10.905 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T03:58:10.945 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T03:58:10.985 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T03:58:11.012 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T03:58:11.051 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:58:11.122 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T03:58:11.149 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T03:58:11.206 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:58:11.206 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T03:58:11.222 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:11.265 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:11.274 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T03:58:11.302 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:11.341 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:11.413 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T03:58:11.441 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T03:58:11.479 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:58:11.651 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T03:58:11.678 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:11.717 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:11.766 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T03:58:11.793 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T03:58:11.832 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:58:11.848 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T03:58:11.877 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:11.970 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T03:58:11.983 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:11.997 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:12.321 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T03:58:12.348 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:12.448 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T03:58:12.465 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:12.583 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:12.953 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T03:58:12.980 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T03:58:13.020 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T03:58:13.057 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph. 2024-09-16T03:58:13.072 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:13.114 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:13.323 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T03:58:13.350 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T03:58:13.389 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:58:13.467 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T03:58:13.494 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:13.534 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:13.718 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T03:58:13.742 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph. 2024-09-16T03:58:13.746 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T03:58:13.769 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:13.808 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:13.809 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:58:13.871 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T03:58:13.898 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T03:58:13.912 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T03:58:13.937 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T03:58:13.939 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:13.979 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:14.086 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T03:58:14.104 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T03:58:14.112 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:14.132 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T03:58:14.144 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:14.171 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T03:58:14.215 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T03:58:14.243 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T03:58:14.281 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package cephadm. 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:14.407 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:14.500 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:14.566 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T03:58:14.572 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:14.576 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T03:58:14.604 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T03:58:14.634 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T03:58:14.754 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T03:58:14.793 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T03:58:14.870 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T03:58:14.870 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T03:58:14.887 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T03:58:14.897 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T03:58:14.921 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:58:14.966 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cephadm. 2024-09-16T03:58:14.994 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:15.033 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:15.079 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T03:58:15.095 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:58:15.137 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:15.206 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T03:58:15.233 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T03:58:15.241 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T03:58:15.269 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T03:58:15.272 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T03:58:15.307 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T03:58:15.336 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T03:58:15.363 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T03:58:15.402 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T03:58:15.584 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T03:58:15.611 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T03:58:15.644 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T03:58:15.650 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:58:15.671 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T03:58:15.688 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T03:58:15.704 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:58:15.710 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:58:15.747 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:16.030 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T03:58:16.057 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T03:58:16.096 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T03:58:16.168 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-babel. 2024-09-16T03:58:16.195 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:58:16.234 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:16.281 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T03:58:16.297 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:16.331 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:16.383 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T03:58:16.410 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T03:58:16.441 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:58:16.562 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T03:58:16.589 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T03:58:16.628 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:58:16.651 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T03:58:16.678 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T03:58:16.717 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:58:16.760 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-babel. 2024-09-16T03:58:16.789 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:58:16.835 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:16.974 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T03:58:17.001 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:17.040 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:17.054 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T03:58:17.081 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:58:17.120 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:58:17.147 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T03:58:17.174 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T03:58:17.213 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:58:17.255 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T03:58:17.283 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:17.321 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:17.348 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package jq. 2024-09-16T03:58:17.375 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:58:17.385 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T03:58:17.413 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T03:58:17.414 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T03:58:17.452 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T03:58:17.566 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T03:58:17.593 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:17.632 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:17.640 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T03:58:17.667 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package socat. 2024-09-16T03:58:17.667 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T03:58:17.685 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T03:58:17.707 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:58:17.713 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-routes. 2024-09-16T03:58:17.725 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:58:17.740 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T03:58:17.788 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:58:17.969 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T03:58:17.993 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T03:58:17.996 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T03:58:18.021 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:58:18.029 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T03:58:18.035 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T03:58:18.051 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:58:18.055 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T03:58:18.066 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T03:58:18.094 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:18.095 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:58:18.125 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:18.296 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package jq. 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:58:18.362 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T03:58:18.433 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-test. 2024-09-16T03:58:18.438 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-routes. 2024-09-16T03:58:18.449 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:18.466 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T03:58:18.491 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:18.505 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:58:18.631 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package socat. 2024-09-16T03:58:18.659 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T03:58:18.698 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:58:18.858 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T03:58:18.898 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:18.924 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:19.000 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T03:58:19.016 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T03:58:19.059 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:58:19.413 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-test. 2024-09-16T03:58:19.441 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:19.480 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T03:58:19.480 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:19.496 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T03:58:19.546 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:58:19.992 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T03:58:20.019 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T03:58:20.058 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:58:20.071 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T03:58:20.099 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T03:58:20.138 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:58:20.395 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T03:58:20.423 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T03:58:20.658 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T03:58:20.658 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:58:20.685 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T03:58:20.724 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:58:20.812 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T03:58:20.840 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:20.878 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:20.973 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T03:58:21.000 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T03:58:21.039 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:58:21.095 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T03:58:21.122 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T03:58:21.161 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:58:21.191 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T03:58:21.207 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:21.241 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:21.414 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T03:58:21.430 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T03:58:21.463 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:58:21.527 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T03:58:21.551 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T03:58:21.585 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:58:21.700 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T03:58:21.707 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T03:58:21.716 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:21.750 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:21.859 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:21.907 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T03:58:21.934 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T03:58:21.973 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:58:22.012 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T03:58:22.028 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:22.062 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:22.157 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T03:58:22.185 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:22.301 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:22.335 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T03:58:22.348 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T03:58:22.363 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T03:58:22.375 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T03:58:22.402 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:58:22.423 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:58:22.744 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T03:58:22.771 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T03:58:22.801 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package pkg-config. 2024-09-16T03:58:22.811 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:58:22.825 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T03:58:22.859 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:58:22.862 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T03:58:22.890 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T03:58:22.929 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T03:58:23.103 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T03:58:23.119 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T03:58:23.153 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:58:23.164 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T03:58:23.173 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T03:58:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T03:58:23.191 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T03:58:23.198 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T03:58:23.207 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T03:58:23.230 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:58:23.232 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:58:23.238 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T03:58:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T03:58:23.536 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T03:58:23.536 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T03:58:23.563 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T03:58:23.573 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T03:58:23.574 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:58:23.600 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T03:58:23.602 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:58:23.618 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package pkg-config. 2024-09-16T03:58:23.630 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T03:58:23.646 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T03:58:23.787 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:58:23.894 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T03:58:23.905 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T03:58:23.917 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T03:58:23.921 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T03:58:23.933 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T03:58:23.945 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T03:58:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:58:23.972 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T03:58:23.983 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T03:58:24.106 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T03:58:24.121 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T03:58:24.163 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:58:24.293 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T03:58:24.304 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T03:58:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T03:58:24.320 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T03:58:24.332 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T03:58:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T03:58:24.359 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T03:58:24.370 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:58:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T03:58:24.532 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T03:58:24.556 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T03:58:24.590 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T03:58:24.675 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T03:58:24.679 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T03:58:24.681 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T03:58:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T03:58:24.706 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T03:58:24.709 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T03:58:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T03:58:24.745 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:58:24.747 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T03:58:24.876 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T03:58:24.904 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T03:58:24.943 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T03:58:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T03:58:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T03:58:25.040 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T03:58:25.057 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T03:58:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:58:25.076 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-py. 2024-09-16T03:58:25.090 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T03:58:25.142 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T03:58:25.288 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T03:58:25.316 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T03:58:25.354 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:58:25.373 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T03:58:25.400 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T03:58:25.402 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T03:58:25.430 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T03:58:25.439 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T03:58:25.462 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T03:58:25.468 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:58:25.490 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T03:58:25.528 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:58:25.666 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T03:58:25.693 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T03:58:25.732 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T03:58:25.742 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T03:58:25.770 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T03:58:25.809 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:58:25.966 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T03:58:25.993 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T03:58:26.041 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:58:26.051 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-py. 2024-09-16T03:58:26.079 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T03:58:26.091 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T03:58:26.118 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T03:58:26.119 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:26.157 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:26.318 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-toml. 2024-09-16T03:58:26.346 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T03:58:26.385 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T03:58:26.438 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T03:58:26.448 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T03:58:26.465 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:26.466 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T03:58:26.477 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T03:58:26.504 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:58:26.505 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T03:58:26.506 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:26.543 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:58:26.688 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T03:58:26.715 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T03:58:26.775 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:58:26.825 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T03:58:26.853 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T03:58:26.855 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T03:58:26.883 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:58:26.892 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:58:26.916 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T03:58:26.921 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:58:26.932 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T03:58:26.966 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:58:27.066 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T03:58:27.093 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T03:58:27.141 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:58:27.166 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package jq. 2024-09-16T03:58:27.183 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:58:27.212 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T03:58:27.224 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T03:58:27.227 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-toml. 2024-09-16T03:58:27.240 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:58:27.255 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T03:58:27.278 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:58:27.294 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T03:58:27.418 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T03:58:27.446 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T03:58:27.477 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package socat. 2024-09-16T03:58:27.484 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:58:27.497 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package jq. 2024-09-16T03:58:27.505 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T03:58:27.514 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:58:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T03:58:27.552 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:58:27.605 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T03:58:27.632 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T03:58:27.671 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:58:27.775 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package socat. 2024-09-16T03:58:27.792 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T03:58:27.833 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:58:27.880 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T03:58:27.897 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T03:58:27.947 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:58:27.967 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T03:58:27.994 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T03:58:28.024 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:58:28.169 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T03:58:28.186 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T03:58:28.228 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:58:28.247 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package radosgw. 2024-09-16T03:58:28.252 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T03:58:28.268 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T03:58:28.275 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:28.302 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:58:28.309 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-test. 2024-09-16T03:58:28.313 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:28.326 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:28.368 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:28.566 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-test. 2024-09-16T03:58:28.582 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:28.624 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:29.006 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package radosgw. 2024-09-16T03:58:29.034 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:29.089 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:29.116 INFO:teuthology.orchestra.run.smithi188.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T03:58:29.132 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:29.166 INFO:teuthology.orchestra.run.smithi188.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:29.438 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T03:58:29.704 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:58:29.813 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:58:29.883 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T03:58:29.900 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:29.941 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:30.055 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T03:58:30.238 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T03:58:30.422 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T03:58:30.471 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:58:30.597 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:58:30.707 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T03:58:30.734 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:30.773 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:30.854 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T03:58:30.862 INFO:teuthology.orchestra.run.smithi188.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T03:58:30.978 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T03:58:31.006 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:58:31.044 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:31.068 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T03:58:31.085 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:31.126 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:31.181 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:31.207 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T03:58:31.323 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T03:58:31.340 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:31.373 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:31.429 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T03:58:31.457 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T03:58:31.496 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:58:31.649 INFO:teuthology.orchestra.run.smithi117.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T03:58:31.676 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T03:58:31.703 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T03:58:31.707 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:31.742 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:58:31.748 INFO:teuthology.orchestra.run.smithi188.stdout:Adding system user cephadm....done 2024-09-16T03:58:31.859 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T03:58:31.871 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:58:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T03:58:31.926 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:58:32.113 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T03:58:32.122 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:58:32.141 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T03:58:32.180 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:58:32.240 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user cephadm....done 2024-09-16T03:58:32.246 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T03:58:32.274 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T03:58:32.313 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:58:32.366 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:58:32.396 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:58:32.491 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:58:32.501 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T03:58:32.528 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T03:58:32.567 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:58:32.656 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:58:32.719 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package pkg-config. 2024-09-16T03:58:32.742 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T03:58:32.747 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T03:58:32.785 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:58:32.890 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:58:32.964 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package pkg-config. 2024-09-16T03:58:32.991 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T03:58:33.002 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:58:33.016 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:58:33.022 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:58:33.046 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T03:58:33.063 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T03:58:33.104 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:58:33.128 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T03:58:33.266 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T03:58:33.267 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T03:58:33.283 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T03:58:33.324 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:58:33.429 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:58:33.526 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:58:33.558 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T03:58:33.586 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T03:58:33.624 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T03:58:33.653 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T03:58:33.694 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T03:58:33.722 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T03:58:33.761 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T03:58:33.773 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T03:58:33.927 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T03:58:33.955 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T03:58:33.962 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:58:33.994 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T03:58:34.033 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T03:58:34.097 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T03:58:34.125 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T03:58:34.163 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T03:58:34.310 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:58:34.314 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T03:58:34.342 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T03:58:34.389 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:58:34.406 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T03:58:34.475 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T03:58:34.503 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T03:58:34.541 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:58:34.577 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:58:34.665 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T03:58:34.701 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T03:58:34.729 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T03:58:34.767 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T03:58:34.836 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:58:34.853 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T03:58:34.881 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T03:58:34.919 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T03:58:34.941 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:58:35.079 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:58:35.103 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-py. 2024-09-16T03:58:35.132 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T03:58:35.170 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T03:58:35.218 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:58:35.256 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-py. 2024-09-16T03:58:35.283 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T03:58:35.305 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:35.330 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T03:58:35.453 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:58:35.499 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T03:58:35.527 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T03:58:35.548 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:58:35.565 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:58:35.650 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T03:58:35.674 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:58:35.678 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T03:58:35.687 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:58:35.717 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:58:35.800 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:58:35.926 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:58:35.929 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:36.036 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T03:58:36.065 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T03:58:36.188 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T03:58:36.193 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:58:36.205 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T03:58:36.237 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:58:36.314 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:58:36.328 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:58:36.440 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:58:36.532 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-toml. 2024-09-16T03:58:36.560 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T03:58:36.566 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:58:36.598 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T03:58:36.631 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-toml. 2024-09-16T03:58:36.642 INFO:teuthology.orchestra.run.smithi188.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T03:58:36.659 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T03:58:36.675 INFO:teuthology.orchestra.run.smithi188.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T03:58:36.692 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:58:36.697 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T03:58:36.730 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:58:36.848 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T03:58:36.901 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T03:58:36.930 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T03:58:36.951 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:58:36.967 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:58:36.975 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T03:58:37.004 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T03:58:37.042 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:58:37.216 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:37.280 INFO:teuthology.orchestra.run.smithi117.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T03:58:37.304 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T03:58:37.325 INFO:teuthology.orchestra.run.smithi117.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T03:58:37.333 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T03:58:37.341 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T03:58:37.370 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T03:58:37.371 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:58:37.387 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:58:37.388 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T03:58:37.428 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:58:37.539 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T03:58:37.617 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:58:37.640 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T03:58:37.657 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T03:58:37.690 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:58:37.700 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T03:58:37.717 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T03:58:37.758 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:58:37.873 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:37.893 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:58:37.999 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T03:58:38.203 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T03:58:38.258 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:58:38.320 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:58:38.327 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package radosgw. 2024-09-16T03:58:38.356 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:38.394 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:38.412 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package radosgw. 2024-09-16T03:58:38.440 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:38.470 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:38.526 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:58:38.811 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T03:58:38.920 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:58:38.997 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:58:39.180 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T03:58:39.196 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:39.238 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:39.257 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:58:39.306 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T03:58:39.323 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:58:39.364 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:39.408 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:58:39.526 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T03:58:39.534 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:58:39.555 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:58:39.661 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T03:58:39.795 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:58:39.823 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:58:39.836 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T03:58:39.903 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:58:39.949 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:58:39.953 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:58:40.054 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:58:40.074 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:58:40.140 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T03:58:40.244 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T03:58:40.310 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T03:58:40.367 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T03:58:40.460 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T03:58:40.478 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:58:40.648 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T03:58:40.735 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T03:58:40.830 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:58:40.862 INFO:teuthology.orchestra.run.smithi037.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T03:58:40.921 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:40.969 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:58:41.098 INFO:teuthology.orchestra.run.smithi039.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T03:58:41.114 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:58:41.155 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:41.338 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:58:41.429 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user cephadm....done 2024-09-16T03:58:41.458 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:58:41.552 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:58:41.638 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user cephadm....done 2024-09-16T03:58:41.639 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:58:41.717 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:58:41.853 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:58:41.868 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:41.886 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:58:41.982 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:58:42.113 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:58:42.154 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:58:42.169 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:58:42.233 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:58:42.239 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:58:42.359 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:42.389 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:58:42.470 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:58:42.499 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T03:58:42.515 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:58:42.652 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:58:42.758 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:58:42.774 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T03:58:42.884 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T03:58:42.953 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:58:43.051 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:58:43.177 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T03:58:43.194 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:58:43.486 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:58:43.554 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T03:58:43.609 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:58:43.839 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T03:58:43.846 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T03:58:43.869 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:43.995 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T03:58:44.098 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T03:58:44.124 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:58:44.141 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:58:44.237 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:58:44.364 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:58:44.365 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:58:44.376 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:44.506 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:58:44.667 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:58:44.684 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:58:44.769 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T03:58:44.808 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:58:44.926 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:58:44.943 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:58:45.020 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:58:45.067 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T03:58:45.146 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:58:45.185 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:58:45.211 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:58:45.272 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:58:45.318 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:45.444 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:58:45.462 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:45.486 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:45.573 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:58:45.696 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:58:45.737 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:58:45.737 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:58:45.814 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:58:45.833 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T03:58:45.864 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:58:45.940 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:58:45.988 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T03:58:45.990 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:58:46.066 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:58:46.083 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:46.116 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:58:46.185 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:58:46.239 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T03:58:46.334 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:58:46.391 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:58:46.461 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:58:46.499 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:58:46.696 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T03:58:46.758 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T03:58:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T03:58:46.798 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T03:58:46.827 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T03:58:46.857 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T03:58:46.861 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:58:46.920 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:58:46.947 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T03:58:46.962 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T03:58:47.001 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:58:47.021 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T03:58:47.126 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T03:58:47.198 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:58:47.330 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:47.381 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:47.402 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:58:47.449 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T03:58:47.456 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T03:58:47.506 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T03:58:47.554 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:58:47.708 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:58:47.724 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:58:47.783 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:58:47.834 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T03:58:47.838 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:58:48.000 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:58:48.050 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:58:48.110 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:58:48.139 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:58:48.228 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:58:48.318 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T03:58:48.352 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T03:58:48.398 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:58:48.427 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:58:48.478 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:58:48.512 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:58:48.524 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:58:48.650 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:48.739 INFO:teuthology.orchestra.run.smithi188.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T03:58:48.847 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:58:48.960 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:58:49.070 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:58:49.114 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:58:49.129 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:58:49.146 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:58:49.240 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:58:49.366 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:49.380 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:58:49.397 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:58:49.456 INFO:teuthology.orchestra.run.smithi117.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T03:58:49.488 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:58:49.506 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:58:49.524 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:58:49.632 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:58:49.633 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:58:49.695 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:58:49.739 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T03:58:49.846 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:58:49.865 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T03:58:49.934 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T03:58:50.105 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T03:58:50.158 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:50.238 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:58:50.268 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:58:50.312 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T03:58:50.360 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T03:58:50.474 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T03:58:50.553 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:58:50.576 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:58:50.600 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T03:58:50.611 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:58:50.851 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:50.877 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:58:50.951 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:58:50.956 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:58:50.995 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:58:51.145 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T03:58:51.244 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:58:51.246 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T03:58:51.247 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:58:51.313 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:58:51.505 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T03:58:51.537 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:58:51.598 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:58:51.698 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:58:51.813 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T03:58:51.841 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:58:51.900 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:58:51.939 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T03:58:51.958 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:58:51.967 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:52.092 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:52.215 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T03:58:52.260 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:58:52.268 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:58:52.410 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:58:52.491 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:58:52.594 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:58:52.645 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T03:58:52.753 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:58:52.828 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:58:52.913 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:58:53.213 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T03:58:53.249 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:58:53.366 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:53.467 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:53.489 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:58:53.585 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:58:53.758 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:58:53.825 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:58:53.921 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:53.968 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:53.993 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:54.016 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:58:54.048 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:54.069 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T03:58:54.173 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T03:58:54.195 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:58:54.260 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:54.369 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T03:58:54.403 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:58:54.450 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:54.529 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:58:54.531 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:54.551 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:54.612 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:58:54.657 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:54.674 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:58:54.685 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:54.713 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:58:54.783 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T03:58:54.830 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:58:54.895 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:58:54.895 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:58:54.982 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:58:55.051 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:55.140 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:58:55.177 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:55.233 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T03:58:55.312 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:55.378 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:55.408 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T03:58:55.479 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:55.484 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:55.526 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:58:55.526 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:58:55.588 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:55.601 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:58:55.669 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:58:55.731 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:58:55.811 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:58:55.886 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:58:56.029 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:56.095 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:58:56.146 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T03:58:56.172 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:56.307 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:56.338 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T03:58:56.405 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T03:58:56.433 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:58:56.580 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T03:58:56.665 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:58:56.840 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:58:56.924 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T03:58:57.001 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:57.099 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T03:58:57.127 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:57.175 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:58:57.253 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T03:58:57.375 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:58:57.379 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:57.501 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T03:58:57.505 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:57.569 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:58:57.654 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:58:57.780 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:57.785 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:58:57.895 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:58:57.906 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:58:57.997 INFO:teuthology.orchestra.run.smithi188.stdout:Adding group ceph....done 2024-09-16T03:58:58.057 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:58.158 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:58:58.289 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:58:58.332 INFO:teuthology.orchestra.run.smithi188.stdout:Adding system user ceph....done 2024-09-16T03:58:58.472 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:58:58.509 INFO:teuthology.orchestra.run.smithi188.stdout:Setting system user ceph properties....done 2024-09-16T03:58:58.527 INFO:teuthology.orchestra.run.smithi188.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T03:58:58.556 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:58:58.642 INFO:teuthology.orchestra.run.smithi117.stdout:Adding group ceph....done 2024-09-16T03:58:58.660 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T03:58:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:58:58.756 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:58:58.808 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:58.881 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T03:58:58.882 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:58:58.967 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T03:58:59.011 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user ceph....done 2024-09-16T03:58:59.017 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:58:59.106 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T03:58:59.118 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:58:59.180 INFO:teuthology.orchestra.run.smithi117.stdout:Setting system user ceph properties....done 2024-09-16T03:58:59.194 INFO:teuthology.orchestra.run.smithi117.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T03:58:59.244 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:58:59.329 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T03:58:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:58:59.461 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:58:59.473 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:59.598 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:58:59.637 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:58:59.683 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T03:58:59.871 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:58:59.905 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T03:59:00.031 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T03:59:00.061 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:59:00.061 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T03:59:00.256 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:59:00.400 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:59:00.507 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:00.524 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:59:00.525 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:00.658 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:59:00.726 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:59:00.747 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:00.865 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T03:59:01.001 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:59:01.027 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:59:01.223 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:59:01.223 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:59:01.287 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T03:59:01.294 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:59:01.360 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:01.412 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T03:59:01.565 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:59:01.565 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:59:01.570 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T03:59:01.671 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T03:59:01.688 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T03:59:01.707 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:01.955 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T03:59:01.956 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:59:02.073 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T03:59:02.113 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:02.232 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:59:02.293 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:59:02.340 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:59:02.340 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:59:02.551 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:02.593 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:59:02.678 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T03:59:02.784 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:59:02.784 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:59:02.838 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:02.988 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:59:03.003 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T03:59:03.107 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:59:03.107 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:59:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:59:03.277 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.332 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:59:03.458 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.494 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:59:03.494 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:59:03.581 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.584 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.623 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:59:03.691 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.693 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:59:03.758 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.825 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.884 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.962 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:03.988 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.010 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:59:04.064 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:59:04.064 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:59:04.122 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.226 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:59:04.226 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:59:04.249 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T03:59:04.338 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.439 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.517 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.565 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T03:59:04.576 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.626 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.719 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.747 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.752 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.845 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.849 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.876 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.971 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:59:04.971 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:59:04.975 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:04.996 INFO:teuthology.orchestra.run.smithi188.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.024 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.093 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.236 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:59:05.236 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:59:05.304 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:59:05.305 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:59:05.382 INFO:teuthology.orchestra.run.smithi188.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T03:59:05.462 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.491 INFO:teuthology.orchestra.run.smithi188.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T03:59:05.589 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.619 INFO:teuthology.orchestra.run.smithi188.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T03:59:05.709 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.714 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.786 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.843 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.848 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:59:05.887 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:05.996 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:06.046 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:06.122 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:59:06.237 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:06.581 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T03:59:06.690 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T03:59:06.744 INFO:teuthology.orchestra.run.smithi188.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T03:59:06.826 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T03:59:07.143 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.244 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.354 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.363 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T03:59:07.386 DEBUG:teuthology.parallel:result is None 2024-09-16T03:59:07.469 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.480 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.570 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.606 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.671 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.789 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:07.843 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T03:59:08.100 INFO:teuthology.orchestra.run.smithi037.stdout:Adding group ceph....done 2024-09-16T03:59:08.340 INFO:teuthology.orchestra.run.smithi039.stdout:Adding group ceph....done 2024-09-16T03:59:08.452 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user ceph....done 2024-09-16T03:59:08.487 INFO:teuthology.orchestra.run.smithi117.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T03:59:08.493 DEBUG:teuthology.parallel:result is None 2024-09-16T03:59:08.571 INFO:teuthology.orchestra.run.smithi037.stdout:Setting system user ceph properties....done 2024-09-16T03:59:08.585 INFO:teuthology.orchestra.run.smithi037.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T03:59:08.625 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user ceph....done 2024-09-16T03:59:08.728 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T03:59:08.751 INFO:teuthology.orchestra.run.smithi039.stdout:Setting system user ceph properties....done 2024-09-16T03:59:08.765 INFO:teuthology.orchestra.run.smithi039.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T03:59:08.929 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T03:59:09.038 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T03:59:09.263 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T03:59:09.492 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:09.610 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:09.756 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:09.899 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:10.067 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:59:10.068 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:59:10.387 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:59:10.387 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:59:10.537 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:10.867 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:10.883 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T03:59:11.230 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T03:59:11.375 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:11.718 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:59:11.718 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:59:11.719 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:11.968 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:59:11.968 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:59:12.199 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:12.440 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:59:12.440 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:59:12.445 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:12.654 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:59:12.654 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:59:12.926 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:13.122 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:13.206 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:59:13.206 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:59:13.409 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:59:13.409 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:59:13.686 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:13.795 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:13.907 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:13.946 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:14.033 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:14.163 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:59:14.164 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:59:14.184 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:59:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:59:14.656 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:14.807 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:14.893 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:14.933 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:15.076 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:15.137 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:59:15.598 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T03:59:15.632 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T03:59:15.705 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T03:59:15.741 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T03:59:15.857 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T03:59:15.894 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T03:59:16.565 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T03:59:17.034 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T03:59:17.225 INFO:teuthology.orchestra.run.smithi037.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T03:59:17.230 DEBUG:teuthology.parallel:result is None 2024-09-16T03:59:17.668 INFO:teuthology.orchestra.run.smithi039.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T03:59:17.672 DEBUG:teuthology.parallel:result is None 2024-09-16T03:59:17.673 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:59:17.817 DEBUG:teuthology.orchestra.run.smithi037:> dpkg-query -W -f '${Version}' ceph 2024-09-16T03:59:17.837 INFO:teuthology.orchestra.run.smithi037.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:59:17.838 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:59:17.838 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T03:59:17.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:59:17.977 DEBUG:teuthology.orchestra.run.smithi039:> dpkg-query -W -f '${Version}' ceph 2024-09-16T03:59:17.997 INFO:teuthology.orchestra.run.smithi039.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:59:17.998 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:59:17.998 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T03:59:18.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:59:18.139 DEBUG:teuthology.orchestra.run.smithi117:> dpkg-query -W -f '${Version}' ceph 2024-09-16T03:59:18.158 INFO:teuthology.orchestra.run.smithi117.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:59:18.159 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:59:18.159 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T03:59:18.161 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:59:18.303 DEBUG:teuthology.orchestra.run.smithi188:> dpkg-query -W -f '${Version}' ceph 2024-09-16T03:59:18.322 INFO:teuthology.orchestra.run.smithi188.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:59:18.322 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:59:18.323 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T03:59:18.324 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T03:59:18.325 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:18.325 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:59:18.340 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:59:18.340 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:59:18.354 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:59:18.354 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:59:18.368 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:59:18.369 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:59:18.382 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T03:59:18.383 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:18.383 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:59:18.397 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:59:18.453 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:59:18.453 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:59:18.469 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:59:18.524 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:59:18.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:59:18.537 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:59:18.590 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:59:18.590 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:59:18.604 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:59:18.657 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T03:59:18.657 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:18.657 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:59:18.671 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:59:18.724 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:59:18.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:59:18.738 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:59:18.793 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:59:18.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:59:18.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:59:18.863 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:59:18.863 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:59:18.877 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:59:18.929 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T03:59:18.930 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:18.930 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:59:18.945 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:59:19.000 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:59:19.000 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:59:19.014 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:59:19.068 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:59:19.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:59:19.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:59:19.134 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:59:19.134 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:59:19.147 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:59:19.201 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T03:59:19.281 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T03:59:19.281 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /var/log/ceph 2024-09-16T03:59:19.284 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2024-09-16T03:59:19.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-09-16T03:59:19.289 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /var/log/ceph 2024-09-16T03:59:19.302 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T03:59:19.302 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:59:19.338 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:59:19.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:59:19.348 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:59:19.361 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T03:59:19.361 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:59:19.398 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:59:19.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:59:19.404 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:59:19.430 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T03:59:19.430 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T03:59:19.430 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926', 'branch': 'squid', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905926', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi037.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9KCYuOYbk0CeaPowp25kaJj7YHOWaP7IJyUKpC+R3+hb0o1glDKlwrK/VGf6oKS1fGQV4rFhCtUbyo4PKoba8=', 'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFtJOv5Tu6JY3oQEQHbuJEnyvd1pZauNX4nuRRytC+gBT/8E2EKqQ/oU6CjndNj+VcUxIMLxUa0ANqOF7S7R6s=', 'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3wo++gdDXP/6ydD3cgPDb79KlgLK6IKEdLgVMSNrMzZqk6S0fhjq/L6XcSBIn4MeGnJkj/eomOa5r0/7N7YIk=', 'smithi188.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOPrh9WaFxQsMRthRp+49kzCOBVBzhFmlvlc1a/UdPRKhO/ZldCnyL6g14sw28Rh6Zhi/8dR4VPUtvJ7HObjAQg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T03:59:19.431 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:59:19.458 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:59:19.460 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:59:19.463 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:59:19.469 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:59:19.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:59:19.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:59:19.516 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:59:19.530 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:19.530 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:59:19.565 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:59:19.565 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-09-16T03:59:19.613 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:59:19.613 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:19.613 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-09-16T03:59:19.613 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:19.614 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-16 03:53:38.724360139 +0000 2024-09-16T03:59:19.614 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-16 03:53:38.472362390 +0000 2024-09-16T03:59:19.614 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-16 03:53:38.472362390 +0000 2024-09-16T03:59:19.614 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-09-16T03:59:19.614 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:59:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-16T03:59:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-16T03:59:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000455633 s, 1.1 MB/s 2024-09-16T03:59:19.669 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:59:19.719 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-09-16T03:59:19.765 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:59:19.766 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:19.766 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-09-16T03:59:19.766 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:19.766 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-16 03:53:39.196355922 +0000 2024-09-16T03:59:19.766 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-16 03:53:38.944358174 +0000 2024-09-16T03:59:19.766 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-16 03:53:38.944358174 +0000 2024-09-16T03:59:19.766 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-09-16T03:59:19.766 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:59:19.820 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-16T03:59:19.820 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-16T03:59:19.820 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000444077 s, 1.2 MB/s 2024-09-16T03:59:19.821 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:59:19.870 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-09-16T03:59:19.917 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:59:19.917 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:19.918 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-09-16T03:59:19.918 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:19.918 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-16 03:53:39.432353814 +0000 2024-09-16T03:59:19.918 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-16 03:53:39.428353850 +0000 2024-09-16T03:59:19.918 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-16 03:53:39.428353850 +0000 2024-09-16T03:59:19.918 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-09-16T03:59:19.918 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:59:19.972 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-16T03:59:19.972 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-16T03:59:19.973 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000363815 s, 1.4 MB/s 2024-09-16T03:59:19.973 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:59:20.023 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-09-16T03:59:20.070 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:59:20.070 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:20.070 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 942 Links: 1 2024-09-16T03:59:20.070 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:20.070 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-16 03:53:40.208346883 +0000 2024-09-16T03:59:20.070 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-16 03:53:39.932349348 +0000 2024-09-16T03:59:20.070 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-16 03:53:39.932349348 +0000 2024-09-16T03:59:20.070 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-09-16T03:59:20.071 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:59:20.124 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-16T03:59:20.124 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-16T03:59:20.124 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.00043218 s, 1.2 MB/s 2024-09-16T03:59:20.126 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:59:20.174 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:20.174 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:59:20.174 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:59:20.181 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:59:20.181 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-09-16T03:59:20.229 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:59:20.230 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:20.230 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 862 Links: 1 2024-09-16T03:59:20.230 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:20.230 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-16 03:53:36.800431634 +0000 2024-09-16T03:59:20.230 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-16 03:53:36.564433742 +0000 2024-09-16T03:59:20.230 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-16 03:53:36.564433742 +0000 2024-09-16T03:59:20.230 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-16T03:59:20.230 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:59:20.285 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-16T03:59:20.285 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-16T03:59:20.285 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000419343 s, 1.2 MB/s 2024-09-16T03:59:20.286 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:59:20.334 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-09-16T03:59:20.381 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:59:20.381 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:20.381 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-09-16T03:59:20.381 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:20.381 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-16 03:53:37.036429526 +0000 2024-09-16T03:59:20.381 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-16 03:53:37.032429562 +0000 2024-09-16T03:59:20.381 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-16 03:53:37.032429562 +0000 2024-09-16T03:59:20.381 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-16T03:59:20.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:59:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-16T03:59:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-16T03:59:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000621729 s, 824 kB/s 2024-09-16T03:59:20.437 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:59:20.486 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-09-16T03:59:20.533 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:59:20.533 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:20.533 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 896 Links: 1 2024-09-16T03:59:20.533 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:20.533 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-16 03:53:37.780422881 +0000 2024-09-16T03:59:20.533 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-16 03:53:37.516425239 +0000 2024-09-16T03:59:20.533 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-16 03:53:37.516425239 +0000 2024-09-16T03:59:20.533 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-16T03:59:20.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:59:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-16T03:59:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-16T03:59:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.00043047 s, 1.2 MB/s 2024-09-16T03:59:20.592 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:59:20.642 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-09-16T03:59:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:59:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:20.689 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-09-16T03:59:20.689 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:20.689 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-16 03:53:38.264418557 +0000 2024-09-16T03:59:20.689 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-16 03:53:38.016420773 +0000 2024-09-16T03:59:20.689 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-16 03:53:38.016420773 +0000 2024-09-16T03:59:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-16T03:59:20.690 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:59:20.743 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-16T03:59:20.743 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-16T03:59:20.743 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000405533 s, 1.3 MB/s 2024-09-16T03:59:20.745 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:59:20.794 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:20.794 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:59:20.794 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:59:20.800 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:59:20.800 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-09-16T03:59:20.848 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:59:20.848 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:20.848 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-09-16T03:59:20.848 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:20.848 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-16 03:53:36.949459259 +0000 2024-09-16T03:59:20.848 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-16 03:53:36.721461295 +0000 2024-09-16T03:59:20.848 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-16 03:53:36.721461295 +0000 2024-09-16T03:59:20.848 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2024-09-16T03:59:20.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:59:20.903 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-16T03:59:20.906 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-16T03:59:20.907 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000447979 s, 1.1 MB/s 2024-09-16T03:59:20.908 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:59:20.957 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-09-16T03:59:21.003 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:59:21.003 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:21.004 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 902 Links: 1 2024-09-16T03:59:21.004 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:21.004 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-16 03:53:37.449454793 +0000 2024-09-16T03:59:21.004 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-16 03:53:37.177457222 +0000 2024-09-16T03:59:21.004 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-16 03:53:37.177457222 +0000 2024-09-16T03:59:21.004 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2024-09-16T03:59:21.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:59:21.058 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-16T03:59:21.058 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-16T03:59:21.059 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000438033 s, 1.2 MB/s 2024-09-16T03:59:21.060 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:59:21.109 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-09-16T03:59:21.156 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:59:21.156 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:21.156 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-16T03:59:21.156 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:21.156 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-16 03:53:37.933450469 +0000 2024-09-16T03:59:21.156 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-16 03:53:37.685452684 +0000 2024-09-16T03:59:21.156 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-16 03:53:37.685452684 +0000 2024-09-16T03:59:21.157 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2024-09-16T03:59:21.157 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:59:21.210 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-16T03:59:21.211 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-16T03:59:21.211 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000389483 s, 1.3 MB/s 2024-09-16T03:59:21.212 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:59:21.261 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-09-16T03:59:21.311 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:59:21.312 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:21.312 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-09-16T03:59:21.312 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:21.312 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-16 03:53:38.417446146 +0000 2024-09-16T03:59:21.312 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-16 03:53:38.177448290 +0000 2024-09-16T03:59:21.312 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-16 03:53:38.177448290 +0000 2024-09-16T03:59:21.312 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2024-09-16T03:59:21.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:59:21.366 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-16T03:59:21.366 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-16T03:59:21.366 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000396887 s, 1.3 MB/s 2024-09-16T03:59:21.367 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:59:21.420 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:21.421 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:59:21.421 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:59:21.427 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:59:21.427 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-09-16T03:59:21.474 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:59:21.474 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:21.474 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-09-16T03:59:21.474 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:21.475 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-09-16 03:53:36.779522812 +0000 2024-09-16T03:59:21.475 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-09-16 03:53:36.535524991 +0000 2024-09-16T03:59:21.475 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-09-16 03:53:36.535524991 +0000 2024-09-16T03:59:21.475 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-09-16T03:59:21.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:59:21.529 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-09-16T03:59:21.529 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-09-16T03:59:21.529 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00045114 s, 1.1 MB/s 2024-09-16T03:59:21.530 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:59:21.580 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-09-16T03:59:21.626 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:59:21.626 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:21.626 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-09-16T03:59:21.626 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:21.626 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-09-16 03:53:37.023520633 +0000 2024-09-16T03:59:21.626 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-09-16 03:53:37.019520667 +0000 2024-09-16T03:59:21.627 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-09-16 03:53:37.019520667 +0000 2024-09-16T03:59:21.627 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-09-16T03:59:21.627 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:59:21.681 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-09-16T03:59:21.681 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-09-16T03:59:21.681 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000433103 s, 1.2 MB/s 2024-09-16T03:59:21.682 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:59:21.731 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-09-16T03:59:21.778 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:59:21.779 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:21.779 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-09-16T03:59:21.779 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:21.779 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-09-16 03:53:37.763514022 +0000 2024-09-16T03:59:21.779 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-09-16 03:53:37.523516167 +0000 2024-09-16T03:59:21.780 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-09-16 03:53:37.523516167 +0000 2024-09-16T03:59:21.780 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-09-16T03:59:21.780 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:59:21.836 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-09-16T03:59:21.836 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-09-16T03:59:21.836 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00041861 s, 1.2 MB/s 2024-09-16T03:59:21.837 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:59:21.887 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-09-16T03:59:21.935 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:59:21.935 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:59:21.935 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 943 Links: 1 2024-09-16T03:59:21.935 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:59:21.935 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-09-16 03:53:38.223509914 +0000 2024-09-16T03:59:21.935 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-09-16 03:53:37.979512094 +0000 2024-09-16T03:59:21.936 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-09-16 03:53:37.979512094 +0000 2024-09-16T03:59:21.936 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-09-16T03:59:21.936 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:59:21.989 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-09-16T03:59:21.989 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-09-16T03:59:21.989 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000388764 s, 1.3 MB/s 2024-09-16T03:59:21.990 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:59:22.039 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:22.040 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T03:59:22.040 INFO:tasks.ceph:Generating config... 2024-09-16T03:59:22.041 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T03:59:22.041 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T03:59:22.041 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T03:59:22.041 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-09-16T03:59:22.041 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T03:59:22.041 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T03:59:22.042 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd heartbeat use min delay socket = True 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-16T03:59:22.043 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T03:59:22.043 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T03:59:22.043 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T03:59:22.044 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T03:59:22.044 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:59:22.044 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T03:59:22.044 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T03:59:22.044 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T03:59:22.044 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T03:59:22.044 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T03:59:22.044 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T03:59:22.044 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T03:59:22.045 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T03:59:22.082 INFO:teuthology.orchestra.run.smithi037.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T03:59:22.091 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T03:59:22.167 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:59:22.220 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.37'), ('mon.b', '172.21.15.39'), ('mon.c', '172.21.15.117')] 2024-09-16T03:59:22.220 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.37,172.21.15.39,172.21.15.117', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T03:59:22.221 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:22.221 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:59:22.269 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.37 --add b 172.21.15.39 --add c 172.21.15.117 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: generated fsid e6b3d5c9-0596-44ca-b271-bebb6e706efc 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:setting min_mon_release = quincy 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 0 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:fsid e6b3d5c9-0596-44ca-b271-bebb6e706efc 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:last_changed 2024-09-16T03:59:22.356935+0000 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-09-16T03:59:22.356935+0000 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:min_mon_release 17 (quincy) 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:election_strategy: 1 2024-09-16T03:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T03:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-16T03:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-09-16T03:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T03:59:22.359 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:59:22.364 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e6b3d5c9-0596-44ca-b271-bebb6e706efc... 2024-09-16T03:59:22.365 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:59:22.410 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:59:22.412 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:59:22.415 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-09-16T03:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout: chdir = "" 2024-09-16T03:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout: auth supported = cephx 2024-09-16T03:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout: filestore xattr use omap = true 2024-09-16T03:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout: mon clock drift allowed = 1.000 2024-09-16T03:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: osd crush chooseleaf type = 0 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: auth debug = true 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on old message = true 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on bug = true 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: mon pg warn max object skew = 0 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default size = 2 2024-09-16T03:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow primary affinity = true 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow pg remap = true 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on no sortbitwise = false 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on too few osds = false 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: mon_allow_pool_size_one = true 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: osd default data pool replay window = 5 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: mon allow pool delete = true 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: mon cluster log file level = debug 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: debug asserts on shutdown = true 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: mon health detail to clog = false 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: mon host = "172.21.15.37,172.21.15.39,172.21.15.117" 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: mon client directed command retry = 5 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: mon election default strategy = 1 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject socket failures = 5000 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = e6b3d5c9-0596-44ca-b271-bebb6e706efc 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:[osd] 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd journal size = 100 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub load threshold = 5.0 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub max interval = 120 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd recover clone overlap = true 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd recovery max chunk = 1048576 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug shutdown = true 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug op order = true 2024-09-16T03:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify stray on activate = true 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug trim objects = true 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: osd open classes on start = true 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug pg log writeout = true 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: osd map max advance = 10 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: journal zero on create = true 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout: filestore apply finisher threads = 3 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: bdev debug aio = true 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug misdirected ops = true 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: bdev async discard = True 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: bdev enable discard = True 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore allocator = stupid 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore block size = 96636764160 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore fsck on mount = True 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore zero block detection = True 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluefs = 20 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluestore = 20 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-16T03:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout: debug osd = 20 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: debug rocksdb = 10 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd full ratio = 0.9 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug reject backfill probability = 0.3 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify cached snaps = True 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify missing on start = True 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd heartbeat use min delay socket = True 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd max backfills = 3 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd max markdown count = 1000 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock override recovery settings = True 2024-09-16T03:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout: osd objectstore = bluestore 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue = debug_random 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue cut off = debug_random 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub min interval = 60 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: osd shutdown pgref assert = True 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr] 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: debug mgr = 20 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: mgr/telemetry/nag = false 2024-09-16T03:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:[mon] 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: debug paxos = 20 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: mon data avail warn = 5 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: mon mgr mkfs grace = 240 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd reporter subtree level = osd 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd prime pg temp = true 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: # 1m isn't quite enough 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub interval = 300 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:[client] 2024-09-16T03:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout: rgw cache enabled = true 2024-09-16T03:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable ops log = true 2024-09-16T03:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable usage log = true 2024-09-16T03:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.a] 2024-09-16T03:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.b] 2024-09-16T03:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.c] 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout:[global] 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout: chdir = "" 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout: auth supported = cephx 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout: filestore xattr use omap = true 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout: mon clock drift allowed = 1.000 2024-09-16T03:59:22.451 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: osd crush chooseleaf type = 0 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: auth debug = true 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on old message = true 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on bug = true 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: mon pg warn max object skew = 0 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.452 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default size = 2 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow primary affinity = true 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow pg remap = true 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on no sortbitwise = false 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on too few osds = false 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: mon_allow_pool_size_one = true 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.453 INFO:teuthology.orchestra.run.smithi188.stdout: osd default data pool replay window = 5 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: mon allow pool delete = true 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: mon cluster log file level = debug 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: debug asserts on shutdown = true 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: mon health detail to clog = false 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: mon host = "172.21.15.37,172.21.15.39,172.21.15.117" 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: mon client directed command retry = 5 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: mon election default strategy = 1 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: ms inject socket failures = 5000 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: fsid = e6b3d5c9-0596-44ca-b271-bebb6e706efc 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.454 INFO:teuthology.orchestra.run.smithi188.stdout:[osd] 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd journal size = 100 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub load threshold = 5.0 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub max interval = 120 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd recover clone overlap = true 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd recovery max chunk = 1048576 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug shutdown = true 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug op order = true 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify stray on activate = true 2024-09-16T03:59:22.455 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug trim objects = true 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: osd open classes on start = true 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug pg log writeout = true 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: osd map max advance = 10 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: journal zero on create = true 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.456 INFO:teuthology.orchestra.run.smithi188.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: filestore apply finisher threads = 3 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: bdev debug aio = true 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug misdirected ops = true 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: debug monc = 20 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: bdev async discard = True 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: bdev enable discard = True 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore allocator = stupid 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore block size = 96636764160 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore fsck on mount = True 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore zero block detection = True 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluefs = 20 2024-09-16T03:59:22.457 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluestore = 20 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: debug osd = 20 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: debug rocksdb = 10 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd full ratio = 0.9 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug reject backfill probability = 0.3 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify cached snaps = True 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify missing on start = True 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: osd heartbeat use min delay socket = True 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: osd max backfills = 3 2024-09-16T03:59:22.458 INFO:teuthology.orchestra.run.smithi188.stdout: osd max markdown count = 1000 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: osd mclock override recovery settings = True 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: osd objectstore = bluestore 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: osd op queue = debug_random 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: osd op queue cut off = debug_random 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub min interval = 60 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: osd shutdown pgref assert = True 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr] 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: debug mgr = 20 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:59:22.459 INFO:teuthology.orchestra.run.smithi188.stdout: mgr/telemetry/nag = false 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout:[mon] 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: debug paxos = 20 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: mon data avail warn = 5 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: mon mgr mkfs grace = 240 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd reporter subtree level = osd 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd prime pg temp = true 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.460 INFO:teuthology.orchestra.run.smithi188.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: # 1m isn't quite enough 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: mon scrub interval = 300 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-09-16T03:59:22.461 INFO:teuthology.orchestra.run.smithi188.stdout:[client] 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi188.stdout: rgw cache enabled = true 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable ops log = true 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable usage log = true 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi188.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi188.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.a] 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.b] 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.c] 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-09-16T03:59:22.462 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-09-16T03:59:22.463 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-09-16T03:59:22.464 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-09-16T03:59:22.465 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "172.21.15.37,172.21.15.39,172.21.15.117" 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: mon client directed command retry = 5 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: mon election default strategy = 1 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: ms inject socket failures = 5000 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = e6b3d5c9-0596-44ca-b271-bebb6e706efc 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.466 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 120 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug trim objects = true 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-09-16T03:59:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: bdev async discard = True 2024-09-16T03:59:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: bdev enable discard = True 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore allocator = stupid 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore zero block detection = True 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 20 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 20 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 10 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:59:22.469 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug reject backfill probability = 0.3 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify cached snaps = True 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify missing on start = True 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd heartbeat use min delay socket = True 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd max backfills = 3 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd max markdown count = 1000 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock override recovery settings = True 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue = debug_random 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue cut off = debug_random 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub min interval = 60 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: osd shutdown pgref assert = True 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-09-16T03:59:22.470 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-09-16T03:59:22.471 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.472 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: mon scrub interval = 300 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2024-09-16T03:59:22.473 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.474 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:59:22.475 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-09-16T03:59:22.476 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.37,172.21.15.39,172.21.15.117" 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: mon client directed command retry = 5 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: mon election default strategy = 1 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject socket failures = 5000 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = e6b3d5c9-0596-44ca-b271-bebb6e706efc 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 120 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:59:22.477 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.478 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: debug monc = 20 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = stupid 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-09-16T03:59:22.479 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore zero block detection = True 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 20 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 20 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 10 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug reject backfill probability = 0.3 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify cached snaps = True 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify missing on start = True 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:59:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: osd heartbeat use min delay socket = True 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: osd max backfills = 3 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: osd max markdown count = 1000 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock override recovery settings = True 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: osd op queue = debug_random 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: osd op queue cut off = debug_random 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub min interval = 60 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: osd shutdown pgref assert = True 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-09-16T03:59:22.481 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-09-16T03:59:22.482 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-09-16T03:59:22.483 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: mon scrub interval = 300 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-09-16T03:59:22.484 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-09-16T03:59:22.487 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T03:59:22.487 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T03:59:22.581 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T03:59:22.581 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:22.581 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T03:59:22.587 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:22.587 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T03:59:22.633 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T03:59:22.633 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:22.633 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:59:22.633 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:59:22.698 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T03:59:22.698 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:22.745 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T03:59:22.745 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:59:22.746 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:59:22.746 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:59:22.769 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T03:59:22.770 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:22.818 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T03:59:22.818 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:59:22.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:59:22.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:59:22.840 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T03:59:22.840 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:22.888 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T03:59:22.888 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:59:22.888 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:59:22.888 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:59:22.913 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T03:59:22.913 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:22.962 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T03:59:22.962 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T03:59:22.962 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T03:59:23.007 INFO:teuthology.orchestra.run.smithi037.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T03:59:23.010 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:59:23.055 INFO:teuthology.orchestra.run.smithi188.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:59:23.069 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T03:59:23.069 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T03:59:23.069 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T03:59:23.147 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T03:59:23.158 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T03:59:23.158 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T03:59:23.158 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T03:59:23.173 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:23.173 INFO:tasks.ceph:role: osd.0 2024-09-16T03:59:23.173 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T03:59:23.173 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-09-16T03:59:23.690 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-16T03:59:23.690 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T03:59:23.778 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T03:59:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:23.788 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:59:23.788 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T03:59:23.843 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:23.843 INFO:tasks.ceph:role: osd.4 2024-09-16T03:59:23.843 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T03:59:23.843 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-09-16T03:59:24.372 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-16T03:59:24.373 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T03:59:24.522 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T03:59:24.533 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:59:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:24.534 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-16T03:59:24.589 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:24.590 INFO:tasks.ceph:role: osd.8 2024-09-16T03:59:24.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T03:59:24.590 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:59:25.144 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-09-16T03:59:25.146 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-16T03:59:25.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-16T03:59:25.292 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-16T03:59:25.303 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:59:25.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:25.304 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-16T03:59:25.361 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:25.361 INFO:tasks.ceph:role: osd.12 2024-09-16T03:59:25.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T03:59:25.361 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:59:25.425 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:59:25.425 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:59:25.425 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:59:25.425 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:59:25.425 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:59:25.425 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:59:25.425 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:59:25.426 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:59:25.426 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:59:25.426 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:59:25.845 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-09-16T03:59:25.847 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-16T03:59:25.847 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-16T03:59:25.937 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-16T03:59:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:25.946 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:59:25.947 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:26.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:26.055+0000 7f064b8cf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T03:59:26.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:26.059+0000 7f064b8cf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T03:59:26.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:26.059+0000 7f064b8cf7c0 -1 bdev(0x5630fa7ed800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T03:59:26.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:26.059+0000 7f064b8cf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T03:59:29.458 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T03:59:29.472 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:29.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:29.551+0000 7ff9aa7967c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T03:59:29.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:29.551+0000 7ff9aa7967c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T03:59:29.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:29.551+0000 7ff9aa7967c0 -1 bdev(0x560b88517800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T03:59:29.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:29.551+0000 7ff9aa7967c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T03:59:32.741 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T03:59:32.797 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:32.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:32.879+0000 7fd7b6a7a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-16T03:59:32.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:32.879+0000 7fd7b6a7a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-16T03:59:32.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:32.879+0000 7fd7b6a7a7c0 -1 bdev(0x559d5c4b9800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-16T03:59:32.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:32.879+0000 7fd7b6a7a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-16T03:59:36.265 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-16T03:59:36.322 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:36.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:36.403+0000 7f837563d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-16T03:59:36.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:36.403+0000 7f837563d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-16T03:59:36.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:36.403+0000 7f837563d7c0 -1 bdev(0x564a956f1800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-16T03:59:36.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T03:59:36.403+0000 7f837563d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-16T03:59:39.781 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-16T03:59:39.838 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T03:59:39.854 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:39.854 INFO:tasks.ceph:role: osd.1 2024-09-16T03:59:39.854 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T03:59:39.854 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:59:39.940 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:59:39.940 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:59:39.940 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:59:39.940 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:59:39.940 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:59:39.940 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:59:39.941 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:59:39.941 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:59:39.941 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:59:39.941 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:59:41.236 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-16T03:59:41.238 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-16T03:59:41.238 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-16T03:59:41.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T03:59:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:41.317 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:59:41.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T03:59:41.374 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:41.374 INFO:tasks.ceph:role: osd.5 2024-09-16T03:59:41.374 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T03:59:41.375 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:59:41.455 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:59:41.456 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:59:42.760 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-16T03:59:42.762 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-16T03:59:42.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T03:59:42.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T03:59:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:42.875 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:59:42.876 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-16T03:59:42.933 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:42.933 INFO:tasks.ceph:role: osd.9 2024-09-16T03:59:42.933 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T03:59:42.933 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:59:43.018 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:59:43.019 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:59:44.322 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-16T03:59:44.324 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-16T03:59:44.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-16T03:59:44.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-16T03:59:44.441 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:59:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:44.442 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-16T03:59:44.497 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:59:44.497 INFO:tasks.ceph:role: osd.13 2024-09-16T03:59:44.497 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T03:59:44.497 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:59:44.582 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:59:44.582 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:59:44.582 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:59:44.582 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:59:44.582 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:59:44.582 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:59:44.583 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:59:44.583 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:59:44.583 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:59:44.583 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:59:45.886 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-16T03:59:45.888 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-16T03:59:45.888 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-16T03:59:45.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-16T03:59:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:46.011 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:59:46.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:46.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:46.121+0000 7f3b3339c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T03:59:46.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:46.121+0000 7f3b3339c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T03:59:46.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:46.121+0000 7f3b3339c7c0 -1 bdev(0x55a60221f800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T03:59:46.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:46.125+0000 7f3b3339c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T03:59:50.065 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T03:59:50.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:50.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:50.153+0000 7f58f5a1f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T03:59:50.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:50.153+0000 7f58f5a1f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T03:59:50.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:50.153+0000 7f58f5a1f7c0 -1 bdev(0x5614eec0d800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T03:59:50.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:50.153+0000 7f58f5a1f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T03:59:53.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T03:59:53.861 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:53.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:53.941+0000 7f154a6f77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-16T03:59:53.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:53.941+0000 7f154a6f77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-16T03:59:53.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:53.941+0000 7f154a6f77c0 -1 bdev(0x556c9735f800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-16T03:59:53.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:53.941+0000 7f154a6f77c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-16T03:59:57.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-16T03:59:57.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:59:57.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:57.469+0000 7f97ac08e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-16T03:59:57.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:57.469+0000 7f97ac08e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-16T03:59:57.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:57.469+0000 7f97ac08e7c0 -1 bdev(0x563c01f27800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-16T03:59:57.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-16T03:59:57.469+0000 7f97ac08e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-16T04:00:00.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-16T04:00:00.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T04:00:00.915 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:00:00.915 INFO:tasks.ceph:role: osd.2 2024-09-16T04:00:00.915 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:00:00.915 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:00:00.979 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:00:00.979 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:00:00.979 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:00:00.979 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:00:00.979 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:00:00.980 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:00:00.980 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:00:00.980 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:00:00.980 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:00:00.980 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:00:01.446 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-16T04:00:01.448 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-16T04:00:01.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-16T04:00:01.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T04:00:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:01.558 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:00:01.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T04:00:01.614 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:00:01.614 INFO:tasks.ceph:role: osd.6 2024-09-16T04:00:01.614 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:00:01.614 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:00:01.679 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:00:01.679 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:00:01.680 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:00:01.680 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:00:01.680 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:00:01.680 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:00:01.680 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:00:01.680 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:00:01.680 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:00:01.680 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:00:02.157 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-16T04:00:02.160 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-16T04:00:02.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-16T04:00:02.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T04:00:02.322 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:00:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:02.323 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-16T04:00:02.379 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:00:02.379 INFO:tasks.ceph:role: osd.10 2024-09-16T04:00:02.379 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:00:02.379 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:00:02.445 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:00:02.446 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:00:02.915 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-16T04:00:02.917 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-16T04:00:02.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-16T04:00:03.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-16T04:00:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:03.091 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:00:03.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-16T04:00:03.150 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:00:03.151 INFO:tasks.ceph:role: osd.14 2024-09-16T04:00:03.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:00:03.151 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:00:03.214 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:00:03.215 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:00:03.635 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-16T04:00:03.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-16T04:00:03.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-16T04:00:03.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-16T04:00:03.738 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:00:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:03.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:03.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:03.853+0000 7fd7af4a27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T04:00:03.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:03.853+0000 7fd7af4a27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T04:00:03.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:03.853+0000 7fd7af4a27c0 -1 bdev(0x56413cdd7800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T04:00:03.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:03.853+0000 7fd7af4a27c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T04:00:07.239 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T04:00:07.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:07.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:07.333+0000 7fd7c37767c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T04:00:07.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:07.333+0000 7fd7c37767c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T04:00:07.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:07.333+0000 7fd7c37767c0 -1 bdev(0x55ecaf0ab800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T04:00:07.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:07.333+0000 7fd7c37767c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T04:00:10.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T04:00:10.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:10.861+0000 7fecb8b127c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:10.861+0000 7fecb8b127c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:10.861+0000 7fecb8b127c0 -1 bdev(0x564e6c355800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:10.861+0000 7fecb8b127c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-16T04:00:14.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-16T04:00:14.063 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:14.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:14.145+0000 7fcae27447c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-16T04:00:14.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:14.145+0000 7fcae27447c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-16T04:00:14.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:14.145+0000 7fcae27447c0 -1 bdev(0x5643f53a5800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-16T04:00:14.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-16T04:00:14.149+0000 7fcae27447c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-16T04:00:17.535 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-16T04:00:17.591 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T04:00:17.606 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:00:17.606 INFO:tasks.ceph:role: osd.3 2024-09-16T04:00:17.606 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:00:17.607 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:00:17.668 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:00:17.668 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:00:17.669 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:00:17.669 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:00:17.669 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:00:17.669 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:00:17.669 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:00:17.669 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:00:17.669 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:00:17.669 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:00:18.123 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-09-16T04:00:18.125 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-09-16T04:00:18.125 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T04:00:18.241 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T04:00:18.251 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:00:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:18.252 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T04:00:18.308 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:00:18.308 INFO:tasks.ceph:role: osd.7 2024-09-16T04:00:18.308 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:00:18.308 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:00:18.370 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:00:18.370 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:00:18.370 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:00:18.370 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:00:18.370 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:00:18.370 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:00:18.370 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:00:18.370 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:00:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:00:18.371 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:00:18.839 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-09-16T04:00:18.841 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-09-16T04:00:18.842 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-16T04:00:19.006 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T04:00:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:19.017 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:00:19.018 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-16T04:00:19.073 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:00:19.074 INFO:tasks.ceph:role: osd.11 2024-09-16T04:00:19.074 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:00:19.074 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:00:19.137 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:00:19.610 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-09-16T04:00:19.612 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-09-16T04:00:19.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-16T04:00:19.742 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-16T04:00:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:19.754 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:00:19.754 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-16T04:00:19.809 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:00:19.810 INFO:tasks.ceph:role: osd.15 2024-09-16T04:00:19.810 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:00:19.810 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:00:19.873 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:00:19.873 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:00:19.873 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:00:19.873 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:00:19.873 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:00:19.873 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:00:19.873 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:00:19.874 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:00:19.874 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:00:19.874 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:00:20.302 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-09-16T04:00:20.305 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-09-16T04:00:20.306 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-16T04:00:20.401 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-16T04:00:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:20.413 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:00:20.414 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:20.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:20.636+0000 7f9b195037c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T04:00:20.640 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:20.636+0000 7f9b195037c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T04:00:20.640 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:20.636+0000 7f9b195037c0 -1 bdev(0x55ae66b05800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T04:00:20.640 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:20.636+0000 7f9b195037c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T04:00:23.803 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T04:00:23.817 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:23.900 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:23.896+0000 7f30621ce7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T04:00:23.901 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:23.900+0000 7f30621ce7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T04:00:23.901 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:23.900+0000 7f30621ce7c0 -1 bdev(0x5641ab693800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T04:00:23.901 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:23.900+0000 7f30621ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T04:00:27.020 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T04:00:27.077 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:27.161 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:27.156+0000 7f78dabde7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-16T04:00:27.162 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:27.160+0000 7f78dabde7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-16T04:00:27.162 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:27.160+0000 7f78dabde7c0 -1 bdev(0x560dd2e59800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-16T04:00:27.162 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:27.160+0000 7f78dabde7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-16T04:00:30.313 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-16T04:00:30.369 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:30.451 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:30.448+0000 7f77540707c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-16T04:00:30.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:30.448+0000 7f77540707c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-16T04:00:30.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:30.448+0000 7f77540707c0 -1 bdev(0x55955c239800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-16T04:00:30.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-09-16T04:00:30.448+0000 7f77540707c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-16T04:00:33.592 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-16T04:00:33.646 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T04:00:33.647 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:33.647 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T04:00:33.662 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:33.662 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T04:00:33.716 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:33.716 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T04:00:33.777 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:33.777 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-16T04:00:33.832 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:33.832 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-16T04:00:33.888 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T04:00:33.888 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T04:00:33.904 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T04:00:33.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T04:00:33.959 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T04:00:33.959 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-16T04:00:34.017 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T04:00:34.017 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-16T04:00:34.073 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T04:00:34.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T04:00:34.088 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T04:00:34.088 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T04:00:34.143 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T04:00:34.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-16T04:00:34.201 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T04:00:34.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-16T04:00:34.255 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:34.255 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T04:00:34.269 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:34.269 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T04:00:34.325 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:34.325 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T04:00:34.381 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:34.381 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-16T04:00:34.437 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:34.437 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-16T04:00:34.492 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:34.493 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T04:00:34.541 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T04:00:34.542 DEBUG:teuthology.orchestra.run.smithi037:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:00:34.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:00:34.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:00:34.554 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.y] 2024-09-16T04:00:34.554 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAbredm+F5zABAAvlkMx6WxRAMXoxvVCLYw0Q== 2024-09-16T04:00:34.554 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.0] 2024-09-16T04:00:34.554 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAeredmo5BzAxAA25DgvsR42YbtjXeWd3jmUg== 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.4] 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAhredm220dIRAAvHA7/2J9KKbLd9UygGWtxw== 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.8] 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAkredmBYuHNBAAVcT5isEhW71yXqHwqFndsA== 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.12] 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAoredmdLEhGBAA3DwBx9J4S0Ud+CPdvPLi7Q== 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.1] 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAyredm/86cBxAAQUusbjtS6Idb2rbDy9DczQ== 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.5] 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA2redm/liICRAAt1M1L0+Y5kOAObrnT6lSnA== 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.9] 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA5redmNNpoOBAAHwWI0OUvQPyv441KMHzDCg== 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.13] 2024-09-16T04:00:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA9redmbnM5HBAARDtyX2KMqh5nrOw85K5xkw== 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.2] 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBDredmw9sOMxAAa9FZOzDk8Oy7lm6BG1VPgQ== 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.6] 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBHredmAYUgFBAAJQGEexU8lia5Km0CiajUXA== 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.10] 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBKredmkFp+MxAAvXK4Dw7+aQ00S+G84aqUqQ== 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.14] 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBOredmFxf/CBAAHT1rhJ6Yz8BPwexaueGQvw== 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.x] 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAbredmltNWAxAAzJwOq3SGHPHbQQQxPT+pZA== 2024-09-16T04:00:34.556 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.3] 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBUredma7wxJhAA1RLCg2Qlgl6xC0hqLK0npg== 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.7] 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBXredmkF7ANRAA7XILy+MX1F89Zwi5uSAeGg== 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.11] 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBbredmDlStCRAAea2dzohDM60GQNHd3GCDnA== 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.15] 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBeredmr4z8GhAA+h1Lyqi9Aa9HDdNtQFaC0Q== 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-09-16T04:00:34.557 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAbredmS47MCBAAM0iTOdhU2PXRHdav/SyCUA== 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.y] 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAbredm+F5zABAAvlkMx6WxRAMXoxvVCLYw0Q== 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAeredmo5BzAxAA25DgvsR42YbtjXeWd3jmUg== 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.4] 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAhredm220dIRAAvHA7/2J9KKbLd9UygGWtxw== 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.8] 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAkredmBYuHNBAAVcT5isEhW71yXqHwqFndsA== 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.12] 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAoredmdLEhGBAA3DwBx9J4S0Ud+CPdvPLi7Q== 2024-09-16T04:00:34.559 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAyredm/86cBxAAQUusbjtS6Idb2rbDy9DczQ== 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.5] 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQA2redm/liICRAAt1M1L0+Y5kOAObrnT6lSnA== 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.9] 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQA5redmNNpoOBAAHwWI0OUvQPyv441KMHzDCg== 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.13] 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQA9redmbnM5HBAARDtyX2KMqh5nrOw85K5xkw== 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBDredmw9sOMxAAa9FZOzDk8Oy7lm6BG1VPgQ== 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.6] 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBHredmAYUgFBAAJQGEexU8lia5Km0CiajUXA== 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.10] 2024-09-16T04:00:34.560 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBKredmkFp+MxAAvXK4Dw7+aQ00S+G84aqUqQ== 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.14] 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBOredmFxf/CBAAHT1rhJ6Yz8BPwexaueGQvw== 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAbredmltNWAxAAzJwOq3SGHPHbQQQxPT+pZA== 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.3] 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBUredma7wxJhAA1RLCg2Qlgl6xC0hqLK0npg== 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.7] 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBXredmkF7ANRAA7XILy+MX1F89Zwi5uSAeGg== 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.11] 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBbredmDlStCRAAea2dzohDM60GQNHd3GCDnA== 2024-09-16T04:00:34.561 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.15] 2024-09-16T04:00:34.562 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBeredmr4z8GhAA+h1Lyqi9Aa9HDdNtQFaC0Q== 2024-09-16T04:00:34.562 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-09-16T04:00:34.562 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAbredmS47MCBAAM0iTOdhU2PXRHdav/SyCUA== 2024-09-16T04:00:34.562 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.y] 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAbredm+F5zABAAvlkMx6WxRAMXoxvVCLYw0Q== 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAeredmo5BzAxAA25DgvsR42YbtjXeWd3jmUg== 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.4] 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAhredm220dIRAAvHA7/2J9KKbLd9UygGWtxw== 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.8] 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAkredmBYuHNBAAVcT5isEhW71yXqHwqFndsA== 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.12] 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAoredmdLEhGBAA3DwBx9J4S0Ud+CPdvPLi7Q== 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAyredm/86cBxAAQUusbjtS6Idb2rbDy9DczQ== 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.5] 2024-09-16T04:00:34.563 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA2redm/liICRAAt1M1L0+Y5kOAObrnT6lSnA== 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.9] 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA5redmNNpoOBAAHwWI0OUvQPyv441KMHzDCg== 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.13] 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA9redmbnM5HBAARDtyX2KMqh5nrOw85K5xkw== 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBDredmw9sOMxAAa9FZOzDk8Oy7lm6BG1VPgQ== 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.6] 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBHredmAYUgFBAAJQGEexU8lia5Km0CiajUXA== 2024-09-16T04:00:34.564 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.10] 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBKredmkFp+MxAAvXK4Dw7+aQ00S+G84aqUqQ== 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.14] 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBOredmFxf/CBAAHT1rhJ6Yz8BPwexaueGQvw== 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAbredmltNWAxAAzJwOq3SGHPHbQQQxPT+pZA== 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBUredma7wxJhAA1RLCg2Qlgl6xC0hqLK0npg== 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.7] 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBXredmkF7ANRAA7XILy+MX1F89Zwi5uSAeGg== 2024-09-16T04:00:34.565 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.11] 2024-09-16T04:00:34.566 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBbredmDlStCRAAea2dzohDM60GQNHd3GCDnA== 2024-09-16T04:00:34.566 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.15] 2024-09-16T04:00:34.566 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBeredmr4z8GhAA+h1Lyqi9Aa9HDdNtQFaC0Q== 2024-09-16T04:00:34.566 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-09-16T04:00:34.566 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAbredmS47MCBAAM0iTOdhU2PXRHdav/SyCUA== 2024-09-16T04:00:34.566 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:00:34.606 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:00:34.608 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:00:34.656 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.659 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.747 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.828 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.833 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.916 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:34.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.003 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.086 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.090 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.093 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.173 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.178 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.256 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.339 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.345 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.349 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.424 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.430 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.509 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.513 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.592 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.597 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.601 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.675 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:00:35.681 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:00:35.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:00:35.759 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.766 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.846 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.850 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.852 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.934 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.937 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:35.979 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:36.022 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:36.024 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:00:36.063 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:00:36.106 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:00:36.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:00:36.156 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T04:00:36.156 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T04:00:36.167 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:00:36.254 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T04:00:36.307 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T04:00:36.321 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:00:36.423 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T04:00:36.434 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T04:00:36.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:00:36.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T04:00:36.553 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:36.556 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:36.558 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:00:36.602 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T04:00:36.603 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:00:36.603 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:00:36.605 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:00:36.606 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:00:36.606 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:00:36.608 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:00:36.608 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:00:36.608 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:00:36.647 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:00:36.648 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T04:00:36.648 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T04:00:36.648 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T04:00:36.652 INFO:tasks.ceph.mgr.y:Started 2024-09-16T04:00:36.652 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T04:00:36.652 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T04:00:36.655 INFO:tasks.ceph.mgr.x:Started 2024-09-16T04:00:36.655 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T04:00:36.655 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T04:00:42.143 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-09-16T04:00:42.163 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T04:00:42.163 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T04:00:42.163 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:42.163 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T04:00:42.174 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:42.174 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T04:00:42.232 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:42.232 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-16T04:00:42.292 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T04:00:42.292 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-16T04:00:42.349 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T04:00:42.350 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T04:00:42.365 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T04:00:42.365 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T04:00:42.407 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:42.404+0000 7f58168eb180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T04:00:42.420 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T04:00:42.420 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-16T04:00:42.476 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-16T04:00:42.476 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-16T04:00:42.498 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:42.494+0000 7ff4983b8180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T04:00:42.531 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T04:00:42.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T04:00:42.546 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T04:00:42.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T04:00:42.602 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T04:00:42.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-16T04:00:42.657 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-16T04:00:42.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-16T04:00:42.713 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:42.713 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T04:00:42.727 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:42.727 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T04:00:42.782 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:42.782 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-16T04:00:42.837 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-09-16T04:00:42.837 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-16T04:00:42.892 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 841234ee-1773-46d0-a547-ee692a129261 0 2024-09-16T04:00:42.987 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:42.984+0000 7f58168eb180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T04:00:43.019 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:43.014+0000 7ff4983b8180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T04:00:43.138 INFO:teuthology.orchestra.run.smithi188.stdout:0 2024-09-16T04:00:43.156 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 7245a6e4-d25d-406d-9ed1-47ef0c8c1078 1 2024-09-16T04:00:43.199 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:43.196+0000 7f58168eb180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T04:00:43.236 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:43.230+0000 7ff4983b8180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T04:00:43.361 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-09-16T04:00:43.379 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 8605b852-95b1-4488-b860-a04e018a9d96 2 2024-09-16T04:00:43.636 INFO:teuthology.orchestra.run.smithi188.stdout:2 2024-09-16T04:00:43.655 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 9457ba25-d686-4f3b-8b67-6db4aed0af99 3 2024-09-16T04:00:43.795 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:43.792+0000 7f58168eb180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T04:00:43.841 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:43.838+0000 7ff4983b8180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T04:00:43.849 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:43.844+0000 7f58168eb180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T04:00:43.852 INFO:teuthology.orchestra.run.smithi188.stdout:3 2024-09-16T04:00:43.870 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 23118253-3204-403c-b58e-c9cf18689a9f 4 2024-09-16T04:00:43.898 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:43.894+0000 7ff4983b8180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T04:00:43.904 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:43.900+0000 7f58168eb180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T04:00:43.953 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:43.950+0000 7ff4983b8180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T04:00:43.984 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:43.980+0000 7f58168eb180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T04:00:44.037 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:44.030+0000 7ff4983b8180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T04:00:44.037 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:44.036+0000 7f58168eb180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T04:00:44.074 INFO:teuthology.orchestra.run.smithi188.stdout:4 2024-09-16T04:00:44.089 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:44.086+0000 7ff4983b8180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T04:00:44.090 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 8ed21a9c-c9d4-454b-ab83-c14538a08a09 5 2024-09-16T04:00:44.116 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:44.112+0000 7f58168eb180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T04:00:44.170 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:44.166+0000 7ff4983b8180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T04:00:44.170 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:44.168+0000 7f58168eb180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T04:00:44.225 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:44.222+0000 7ff4983b8180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T04:00:44.314 INFO:teuthology.orchestra.run.smithi188.stdout:5 2024-09-16T04:00:44.326 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:44.324+0000 7f58168eb180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T04:00:44.330 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new fc4474a6-228a-430e-a7f6-75c86bb7b514 6 2024-09-16T04:00:44.383 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:44.378+0000 7ff4983b8180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T04:00:44.389 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:44.388+0000 7f58168eb180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T04:00:44.449 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:44.442+0000 7ff4983b8180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T04:00:44.529 INFO:teuthology.orchestra.run.smithi188.stdout:6 2024-09-16T04:00:44.546 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new fb4131d0-f4b4-44bf-a88f-832b545f817e 7 2024-09-16T04:00:44.795 INFO:teuthology.orchestra.run.smithi188.stdout:7 2024-09-16T04:00:44.811 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 55467173-0628-4162-8d2f-fb76319de849 8 2024-09-16T04:00:45.002 INFO:teuthology.orchestra.run.smithi188.stdout:8 2024-09-16T04:00:45.018 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 66db6afc-248e-4197-a30b-2bc766b6b900 9 2024-09-16T04:00:45.047 INFO:tasks.ceph.mgr.x.smithi188.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T04:00:45.047 INFO:tasks.ceph.mgr.x.smithi188.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T04:00:45.047 INFO:tasks.ceph.mgr.x.smithi188.stderr: from numpy import show_config as show_numpy_config 2024-09-16T04:00:45.091 INFO:tasks.ceph.mgr.y.smithi037.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T04:00:45.091 INFO:tasks.ceph.mgr.y.smithi037.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T04:00:45.091 INFO:tasks.ceph.mgr.y.smithi037.stderr: from numpy import show_config as show_numpy_config 2024-09-16T04:00:45.116 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:45.112+0000 7f58168eb180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T04:00:45.160 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:45.154+0000 7ff4983b8180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T04:00:45.211 INFO:teuthology.orchestra.run.smithi188.stdout:9 2024-09-16T04:00:45.227 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 54464474-a85c-4a44-8c78-ca3439d56cf4 10 2024-09-16T04:00:45.290 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:45.288+0000 7f58168eb180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:00:45.347 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:45.342+0000 7ff4983b8180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:00:45.453 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:45.448+0000 7f58168eb180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T04:00:45.477 INFO:teuthology.orchestra.run.smithi188.stdout:10 2024-09-16T04:00:45.493 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 9dabcd8b-28a8-43b2-9636-e3bb7d7a6904 11 2024-09-16T04:00:45.517 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:45.514+0000 7ff4983b8180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T04:00:45.521 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:45.520+0000 7f58168eb180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T04:00:45.591 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:45.586+0000 7ff4983b8180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T04:00:45.669 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:45.664+0000 7f58168eb180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:00:45.724 INFO:teuthology.orchestra.run.smithi188.stdout:11 2024-09-16T04:00:45.741 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 43465856-0b63-41ca-b5e5-57771dbfbe5b 12 2024-09-16T04:00:45.741 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:45.738+0000 7ff4983b8180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:00:45.938 INFO:teuthology.orchestra.run.smithi188.stdout:12 2024-09-16T04:00:45.955 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 7fba1ed8-6948-4b3d-afbf-236c93b9e3e5 13 2024-09-16T04:00:46.150 INFO:teuthology.orchestra.run.smithi188.stdout:13 2024-09-16T04:00:46.167 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 5073cd26-fa48-45c5-ba56-3b0c20878c2a 14 2024-09-16T04:00:46.296 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:46.292+0000 7f58168eb180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T04:00:46.362 INFO:teuthology.orchestra.run.smithi188.stdout:14 2024-09-16T04:00:46.373 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:46.368+0000 7f58168eb180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T04:00:46.378 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 52f9cbf1-4411-47e5-a8e9-442f99215ee3 15 2024-09-16T04:00:46.422 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:46.418+0000 7ff4983b8180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T04:00:46.501 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:46.498+0000 7ff4983b8180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T04:00:46.560 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:46.556+0000 7f58168eb180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T04:00:46.576 INFO:teuthology.orchestra.run.smithi188.stdout:15 2024-09-16T04:00:46.592 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T04:00:46.592 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T04:00:46.595 INFO:tasks.ceph.osd.0:Started 2024-09-16T04:00:46.595 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T04:00:46.595 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T04:00:46.597 INFO:tasks.ceph.osd.4:Started 2024-09-16T04:00:46.597 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-16T04:00:46.597 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-16T04:00:46.599 INFO:tasks.ceph.osd.8:Started 2024-09-16T04:00:46.600 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-16T04:00:46.600 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-16T04:00:46.602 INFO:tasks.ceph.osd.12:Started 2024-09-16T04:00:46.602 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T04:00:46.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T04:00:46.605 INFO:tasks.ceph.osd.1:Started 2024-09-16T04:00:46.605 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T04:00:46.605 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T04:00:46.607 INFO:tasks.ceph.osd.5:Started 2024-09-16T04:00:46.607 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-16T04:00:46.607 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-16T04:00:46.611 INFO:tasks.ceph.osd.9:Started 2024-09-16T04:00:46.611 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-16T04:00:46.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-16T04:00:46.615 INFO:tasks.ceph.osd.13:Started 2024-09-16T04:00:46.615 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T04:00:46.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T04:00:46.618 INFO:tasks.ceph.osd.2:Started 2024-09-16T04:00:46.619 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T04:00:46.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T04:00:46.622 INFO:tasks.ceph.osd.6:Started 2024-09-16T04:00:46.622 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-16T04:00:46.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-16T04:00:46.625 INFO:tasks.ceph.osd.10:Started 2024-09-16T04:00:46.625 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-16T04:00:46.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-16T04:00:46.628 INFO:tasks.ceph.osd.14:Started 2024-09-16T04:00:46.628 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T04:00:46.628 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T04:00:46.630 INFO:tasks.ceph.osd.3:Started 2024-09-16T04:00:46.630 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T04:00:46.630 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T04:00:46.633 INFO:tasks.ceph.osd.7:Started 2024-09-16T04:00:46.633 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-16T04:00:46.633 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-16T04:00:46.636 INFO:tasks.ceph.osd.11:Started 2024-09-16T04:00:46.636 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-16T04:00:46.636 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-16T04:00:46.637 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:46.631+0000 7f58168eb180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T04:00:46.638 INFO:tasks.ceph.osd.15:Started 2024-09-16T04:00:46.639 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:00:46.710 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:46.706+0000 7ff4983b8180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T04:00:46.792 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:46.786+0000 7ff4983b8180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T04:00:46.829 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:00:46.829 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":18,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:00:46.572321+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:00:46.845 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T04:00:46.846 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T04:00:46.944 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:46.939+0000 7f58168eb180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T04:00:47.006 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:47.003+0000 7f58168eb180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T04:00:47.102 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:47.098+0000 7ff4983b8180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T04:00:47.167 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-09-16T04:00:47.162+0000 7ff4983b8180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T04:00:47.814 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:00:47.810+0000 7fd77ef047c0 -1 Falling back to public interface 2024-09-16T04:00:48.004 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:00:48.000+0000 7ffb1c06c7c0 -1 Falling back to public interface 2024-09-16T04:00:48.068 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:48.063+0000 7f57bd520640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T04:00:48.069 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-09-16T04:00:48.067+0000 7f57bd520640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T04:00:48.237 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:00:48.232+0000 7f3e3b7ce7c0 -1 Falling back to public interface 2024-09-16T04:00:48.295 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:00:48.291+0000 7fb92ea3f7c0 -1 Falling back to public interface 2024-09-16T04:00:48.296 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:00:48.291+0000 7f2b6ee847c0 -1 Falling back to public interface 2024-09-16T04:00:48.314 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:00:48.312+0000 7f5853abc7c0 -1 Falling back to public interface 2024-09-16T04:00:48.455 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:00:48.464 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:00:48.460+0000 7f6a756557c0 -1 Falling back to public interface 2024-09-16T04:00:48.475 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:00:48.471+0000 7f731f23a7c0 -1 Falling back to public interface 2024-09-16T04:00:48.476 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:00:48.472+0000 7f85977287c0 -1 Falling back to public interface 2024-09-16T04:00:48.478 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:00:48.476+0000 7fb4112da7c0 -1 Falling back to public interface 2024-09-16T04:00:48.479 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:00:48.475+0000 7f6d01b677c0 -1 Falling back to public interface 2024-09-16T04:00:48.515 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:00:48.510+0000 7fd8319747c0 -1 Falling back to public interface 2024-09-16T04:00:48.534 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:00:48.532+0000 7f65593e77c0 -1 Falling back to public interface 2024-09-16T04:00:48.597 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:00:48.594+0000 7fb366dca7c0 -1 Falling back to public interface 2024-09-16T04:00:48.597 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:00:48.594+0000 7f5c2360c7c0 -1 Falling back to public interface 2024-09-16T04:00:48.714 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:00:48.712+0000 7fe372fa77c0 -1 Falling back to public interface 2024-09-16T04:00:48.759 INFO:teuthology.misc.health.smithi037.stdout: 2024-09-16T04:00:48.759 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":18,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:00:46.572321+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:00:48.775 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-16T04:00:52.241 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:00:52.234+0000 7fd77ef047c0 -1 osd.4 0 log_to_monitors true 2024-09-16T04:00:52.512 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:00:52.507+0000 7f2b6ee847c0 -1 osd.7 0 log_to_monitors true 2024-09-16T04:00:52.576 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:00:52.572+0000 7ffb1c06c7c0 -1 osd.6 0 log_to_monitors true 2024-09-16T04:00:52.576 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:00:52.572+0000 7f3e3b7ce7c0 -1 osd.14 0 log_to_monitors true 2024-09-16T04:00:52.691 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:00:52.687+0000 7f731f23a7c0 -1 osd.15 0 log_to_monitors true 2024-09-16T04:00:52.733 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:00:52.732+0000 7f65593e77c0 -1 osd.13 0 log_to_monitors true 2024-09-16T04:00:52.750 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:00:52.748+0000 7f5853abc7c0 -1 osd.9 0 log_to_monitors true 2024-09-16T04:00:52.753 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:00:52.750+0000 7fd8319747c0 -1 osd.8 0 log_to_monitors true 2024-09-16T04:00:52.837 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:00:52.831+0000 7fb92ea3f7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T04:00:52.923 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:00:52.920+0000 7fe372fa77c0 -1 osd.5 0 log_to_monitors true 2024-09-16T04:00:52.958 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:00:52.955+0000 7f6d01b677c0 -1 osd.11 0 log_to_monitors true 2024-09-16T04:00:53.002 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:00:52.996+0000 7f85977287c0 -1 osd.10 0 log_to_monitors true 2024-09-16T04:00:53.010 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:00:53.004+0000 7f6a756557c0 -1 osd.2 0 log_to_monitors true 2024-09-16T04:00:53.098 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:00:53.094+0000 7f5c2360c7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T04:00:53.152 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:00:53.148+0000 7fb4112da7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T04:00:53.254 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:00:53.250+0000 7fb366dca7c0 -1 osd.12 0 log_to_monitors true 2024-09-16T04:00:54.236 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:00:54.230+0000 7fd77a68a640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T04:00:54.277 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:00:54.274+0000 7fd772cd1640 -1 osd.4 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:54.312 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:00:54.306+0000 7fd82d0fa640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T04:00:54.314 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:00:54.310+0000 7fb362550640 -1 osd.12 0 waiting for initial osdmap 2024-09-16T04:00:54.318 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:00:54.315+0000 7f731a9c0640 -1 osd.15 0 waiting for initial osdmap 2024-09-16T04:00:54.318 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:00:54.315+0000 7f6cfd2ed640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T04:00:54.339 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:00:54.334+0000 7fd829749640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:54.339 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:00:54.334+0000 7fb35eb9f640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:54.349 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:00:54.343+0000 7f731700f640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:54.350 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:00:54.347+0000 7f6cf993c640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:54.356 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:00:54.352+0000 7f584f242640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T04:00:54.356 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:00:54.352+0000 7fb40ca60640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T04:00:54.397 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:00:54.396+0000 7fb4090af640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:54.397 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:00:54.396+0000 7f584b891640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:55.430 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:00:55.426+0000 7f5c1ed92640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T04:00:55.454 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:00:55.450+0000 7f5c173d9640 -1 osd.0 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:55.678 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:00:55.675+0000 7fb92b1fa640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T04:00:55.712 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:00:55.707+0000 7f2b6a60a640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T04:00:55.718 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:00:55.715+0000 7fb92280c640 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:55.746 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:00:55.743+0000 7f2b62c51640 -1 osd.7 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:55.841 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:00:55.840+0000 7f6555ba2640 -1 osd.13 0 waiting for initial osdmap 2024-09-16T04:00:55.851 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:00:55.848+0000 7fe36e72d640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T04:00:55.865 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:00:55.864+0000 7f654d1b4640 -1 osd.13 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:55.885 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:00:55.884+0000 7fe366d74640 -1 osd.5 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:56.383 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:00:56.554 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:00:56.548+0000 7ffb177f2640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T04:00:56.584 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:00:56.580+0000 7ffb0fe39640 -1 osd.6 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:56.627 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:00:56.624+0000 7f8592eae640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T04:00:56.660 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:00:56.656+0000 7f858b4f5640 -1 osd.10 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:56.674 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:00:56.668+0000 7f3e36f54640 -1 osd.14 0 waiting for initial osdmap 2024-09-16T04:00:56.692 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:00:56.688+0000 7f6a70ddb640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T04:00:56.700 INFO:teuthology.misc.health.smithi037.stdout: 2024-09-16T04:00:56.701 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":23,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:00:56.396621+0000","last_up_change":"2024-09-16T04:00:56.396621+0000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6817","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6819","nonce":3422299298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6824","nonce":3422299298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6831","nonce":3422299298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6829","nonce":3422299298}]},"public_addr":"172.21.15.37:6819/3422299298","cluster_addr":"172.21.15.37:6824/3422299298","heartbeat_back_addr":"172.21.15.37:6831/3422299298","heartbeat_front_addr":"172.21.15.37:6829/3422299298","state":["exists","up"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6809","nonce":1856047451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6811","nonce":1856047451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6815","nonce":1856047451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6813","nonce":1856047451}]},"public_addr":"172.21.15.39:6809/1856047451","cluster_addr":"172.21.15.39:6811/1856047451","heartbeat_back_addr":"172.21.15.39:6815/1856047451","heartbeat_front_addr":"172.21.15.39:6813/1856047451","state":["exists","up"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6803","nonce":2905616722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6807","nonce":2905616722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6816","nonce":2905616722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6811","nonce":2905616722}]},"public_addr":"172.21.15.188:6803/2905616722","cluster_addr":"172.21.15.188:6807/2905616722","heartbeat_back_addr":"172.21.15.188:6816/2905616722","heartbeat_front_addr":"172.21.15.188:6811/2905616722","state":["exists","up"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6801","nonce":2698627343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6803","nonce":2698627343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6807","nonce":2698627343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6805","nonce":2698627343}]},"public_addr":"172.21.15.37:6801/2698627343","cluster_addr":"172.21.15.37:6803/2698627343","heartbeat_back_addr":"172.21.15.37:6807/2698627343","heartbeat_front_addr":"172.21.15.37:6805/2698627343","state":["exists","up"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6825","nonce":2772028538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6827","nonce":2772028538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6831","nonce":2772028538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6829","nonce":2772028538}]},"public_addr":"172.21.15.39:6825/2772028538","cluster_addr":"172.21.15.39:6827/2772028538","heartbeat_back_addr":"172.21.15.39:6831/2772028538","heartbeat_front_addr":"172.21.15.39:6829/2772028538","state":["exists","up"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6806","nonce":4221150049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6810","nonce":4221150049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6815","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6817","nonce":4221150049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6813","nonce":4221150049}]},"public_addr":"172.21.15.188:6806/4221150049","cluster_addr":"172.21.15.188:6810/4221150049","heartbeat_back_addr":"172.21.15.188:6817/4221150049","heartbeat_front_addr":"172.21.15.188:6813/4221150049","state":["exists","up"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6809","nonce":3581510146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6811","nonce":3581510146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6815","nonce":3581510146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6813","nonce":3581510146}]},"public_addr":"172.21.15.37:6809/3581510146","cluster_addr":"172.21.15.37:6811/3581510146","heartbeat_back_addr":"172.21.15.37:6815/3581510146","heartbeat_front_addr":"172.21.15.37:6813/3581510146","state":["exists","up"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6801","nonce":2726119652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6803","nonce":2726119652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6807","nonce":2726119652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6805","nonce":2726119652}]},"public_addr":"172.21.15.39:6801/2726119652","cluster_addr":"172.21.15.39:6803/2726119652","heartbeat_back_addr":"172.21.15.39:6807/2726119652","heartbeat_front_addr":"172.21.15.39:6805/2726119652","state":["exists","up"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6823","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6824","nonce":1355328456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6828","nonce":1355328456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6833","nonce":1355328456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6831","nonce":1355328456}]},"public_addr":"172.21.15.188:6824/1355328456","cluster_addr":"172.21.15.188:6828/1355328456","heartbeat_back_addr":"172.21.15.188:6833/1355328456","heartbeat_front_addr":"172.21.15.188:6831/1355328456","state":["exists","up"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6818","nonce":3693386273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6821","nonce":3693386273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6827","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6828","nonce":3693386273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6825","nonce":3693386273}]},"public_addr":"172.21.15.37:6818/3693386273","cluster_addr":"172.21.15.37:6821/3693386273","heartbeat_back_addr":"172.21.15.37:6828/3693386273","heartbeat_front_addr":"172.21.15.37:6825/3693386273","state":["exists","up"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6817","nonce":2905392027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6819","nonce":2905392027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6823","nonce":2905392027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6821","nonce":2905392027}]},"public_addr":"172.21.15.39:6817/2905392027","cluster_addr":"172.21.15.39:6819/2905392027","heartbeat_back_addr":"172.21.15.39:6823/2905392027","heartbeat_front_addr":"172.21.15.39:6821/2905392027","state":["exists","up"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6819","nonce":3000701187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6821","nonce":3000701187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6829","nonce":3000701187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6825","nonce":3000701187}]},"public_addr":"172.21.15.188:6819/3000701187","cluster_addr":"172.21.15.188:6821/3000701187","heartbeat_back_addr":"172.21.15.188:6829/3000701187","heartbeat_front_addr":"172.21.15.188:6825/3000701187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.131369+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.213442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.725374+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.710839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.928910+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.261435+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.711709+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:00:56.711 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:00:56.708+0000 7f3e2f59b640 -1 osd.14 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:00:56.719 DEBUG:teuthology.misc:12 of 16 OSDs are up 2024-09-16T04:00:56.723 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:00:56.720+0000 7f6a69422640 -1 osd.2 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:01:04.330 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:01:04.646 INFO:teuthology.misc.health.smithi037.stdout: 2024-09-16T04:01:04.646 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":26,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:00:59.444891+0000","last_up_change":"2024-09-16T04:00:57.414587+0000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:00:57.133339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6817","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6819","nonce":3422299298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6824","nonce":3422299298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6831","nonce":3422299298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6829","nonce":3422299298}]},"public_addr":"172.21.15.37:6819/3422299298","cluster_addr":"172.21.15.37:6824/3422299298","heartbeat_back_addr":"172.21.15.37:6831/3422299298","heartbeat_front_addr":"172.21.15.37:6829/3422299298","state":["exists","up"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6809","nonce":1856047451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6811","nonce":1856047451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6815","nonce":1856047451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6813","nonce":1856047451}]},"public_addr":"172.21.15.39:6809/1856047451","cluster_addr":"172.21.15.39:6811/1856047451","heartbeat_back_addr":"172.21.15.39:6815/1856047451","heartbeat_front_addr":"172.21.15.39:6813/1856047451","state":["exists","up"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6817","nonce":3343342478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6819","nonce":3343342478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6823","nonce":3343342478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6821","nonce":3343342478}]},"public_addr":"172.21.15.117:6817/3343342478","cluster_addr":"172.21.15.117:6819/3343342478","heartbeat_back_addr":"172.21.15.117:6823/3343342478","heartbeat_front_addr":"172.21.15.117:6821/3343342478","state":["exists","up"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6803","nonce":2905616722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6807","nonce":2905616722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6816","nonce":2905616722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6811","nonce":2905616722}]},"public_addr":"172.21.15.188:6803/2905616722","cluster_addr":"172.21.15.188:6807/2905616722","heartbeat_back_addr":"172.21.15.188:6816/2905616722","heartbeat_front_addr":"172.21.15.188:6811/2905616722","state":["exists","up"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6801","nonce":2698627343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6803","nonce":2698627343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6807","nonce":2698627343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6805","nonce":2698627343}]},"public_addr":"172.21.15.37:6801/2698627343","cluster_addr":"172.21.15.37:6803/2698627343","heartbeat_back_addr":"172.21.15.37:6807/2698627343","heartbeat_front_addr":"172.21.15.37:6805/2698627343","state":["exists","up"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6825","nonce":2772028538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6827","nonce":2772028538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6831","nonce":2772028538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6829","nonce":2772028538}]},"public_addr":"172.21.15.39:6825/2772028538","cluster_addr":"172.21.15.39:6827/2772028538","heartbeat_back_addr":"172.21.15.39:6831/2772028538","heartbeat_front_addr":"172.21.15.39:6829/2772028538","state":["exists","up"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6801","nonce":3124837418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6803","nonce":3124837418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6807","nonce":3124837418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6805","nonce":3124837418}]},"public_addr":"172.21.15.117:6801/3124837418","cluster_addr":"172.21.15.117:6803/3124837418","heartbeat_back_addr":"172.21.15.117:6807/3124837418","heartbeat_front_addr":"172.21.15.117:6805/3124837418","state":["exists","up"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6806","nonce":4221150049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6810","nonce":4221150049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6815","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6817","nonce":4221150049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6813","nonce":4221150049}]},"public_addr":"172.21.15.188:6806/4221150049","cluster_addr":"172.21.15.188:6810/4221150049","heartbeat_back_addr":"172.21.15.188:6817/4221150049","heartbeat_front_addr":"172.21.15.188:6813/4221150049","state":["exists","up"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6809","nonce":3581510146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6811","nonce":3581510146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6815","nonce":3581510146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6813","nonce":3581510146}]},"public_addr":"172.21.15.37:6809/3581510146","cluster_addr":"172.21.15.37:6811/3581510146","heartbeat_back_addr":"172.21.15.37:6815/3581510146","heartbeat_front_addr":"172.21.15.37:6813/3581510146","state":["exists","up"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6801","nonce":2726119652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6803","nonce":2726119652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6807","nonce":2726119652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6805","nonce":2726119652}]},"public_addr":"172.21.15.39:6801/2726119652","cluster_addr":"172.21.15.39:6803/2726119652","heartbeat_back_addr":"172.21.15.39:6807/2726119652","heartbeat_front_addr":"172.21.15.39:6805/2726119652","state":["exists","up"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6825","nonce":737103121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6827","nonce":737103121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6831","nonce":737103121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6829","nonce":737103121}]},"public_addr":"172.21.15.117:6825/737103121","cluster_addr":"172.21.15.117:6827/737103121","heartbeat_back_addr":"172.21.15.117:6831/737103121","heartbeat_front_addr":"172.21.15.117:6829/737103121","state":["exists","up"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6823","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6824","nonce":1355328456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6828","nonce":1355328456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6833","nonce":1355328456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6831","nonce":1355328456}]},"public_addr":"172.21.15.188:6824/1355328456","cluster_addr":"172.21.15.188:6828/1355328456","heartbeat_back_addr":"172.21.15.188:6833/1355328456","heartbeat_front_addr":"172.21.15.188:6831/1355328456","state":["exists","up"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6818","nonce":3693386273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6821","nonce":3693386273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6827","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6828","nonce":3693386273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6825","nonce":3693386273}]},"public_addr":"172.21.15.37:6818/3693386273","cluster_addr":"172.21.15.37:6821/3693386273","heartbeat_back_addr":"172.21.15.37:6828/3693386273","heartbeat_front_addr":"172.21.15.37:6825/3693386273","state":["exists","up"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6817","nonce":2905392027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6819","nonce":2905392027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6823","nonce":2905392027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6821","nonce":2905392027}]},"public_addr":"172.21.15.39:6817/2905392027","cluster_addr":"172.21.15.39:6819/2905392027","heartbeat_back_addr":"172.21.15.39:6823/2905392027","heartbeat_front_addr":"172.21.15.39:6821/2905392027","state":["exists","up"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6809","nonce":1375514506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6811","nonce":1375514506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6815","nonce":1375514506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6813","nonce":1375514506}]},"public_addr":"172.21.15.117:6809/1375514506","cluster_addr":"172.21.15.117:6811/1375514506","heartbeat_back_addr":"172.21.15.117:6815/1375514506","heartbeat_front_addr":"172.21.15.117:6813/1375514506","state":["exists","up"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6819","nonce":3000701187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6821","nonce":3000701187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6829","nonce":3000701187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6825","nonce":3000701187}]},"public_addr":"172.21.15.188:6819/3000701187","cluster_addr":"172.21.15.188:6821/3000701187","heartbeat_back_addr":"172.21.15.188:6829/3000701187","heartbeat_front_addr":"172.21.15.188:6825/3000701187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.056258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.131369+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.987287+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.868903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.213442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.953995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.598108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.496609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.725374+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.710839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.990973+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.928910+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.261435+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.781696+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.550782+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.711709+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:01:04.663 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-16T04:01:04.664 INFO:tasks.ceph:Creating RBD pool 2024-09-16T04:01:04.664 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T04:01:05.707 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'rbd' created 2024-09-16T04:01:05.735 DEBUG:teuthology.orchestra.run.smithi037:> rbd --cluster ceph pool init rbd 2024-09-16T04:01:08.764 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T04:01:08.765 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T04:01:08.765 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T04:01:08.765 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T04:01:08.766 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T04:01:09.126 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:09.144 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":838895118},{"type":"v1","addr":"172.21.15.188:6801","nonce":838895118}]},"active_addr":"172.21.15.188:6801/838895118","active_change":"2024-09-16T04:00:47.027059+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4112,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":1079479459}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":2444120344}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":3480169953}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":707518523}]}]} 2024-09-16T04:01:09.150 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T04:01:09.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:01:09.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:01:09.461 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:09.461 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":30,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:01:08.729364+0000","last_up_change":"2024-09-16T04:00:57.414587+0000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:00:57.133339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:01:05.007470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6817","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6819","nonce":3422299298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6824","nonce":3422299298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6831","nonce":3422299298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6829","nonce":3422299298}]},"public_addr":"172.21.15.37:6819/3422299298","cluster_addr":"172.21.15.37:6824/3422299298","heartbeat_back_addr":"172.21.15.37:6831/3422299298","heartbeat_front_addr":"172.21.15.37:6829/3422299298","state":["exists","up"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6809","nonce":1856047451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6811","nonce":1856047451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6815","nonce":1856047451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6813","nonce":1856047451}]},"public_addr":"172.21.15.39:6809/1856047451","cluster_addr":"172.21.15.39:6811/1856047451","heartbeat_back_addr":"172.21.15.39:6815/1856047451","heartbeat_front_addr":"172.21.15.39:6813/1856047451","state":["exists","up"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6817","nonce":3343342478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6819","nonce":3343342478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6823","nonce":3343342478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6821","nonce":3343342478}]},"public_addr":"172.21.15.117:6817/3343342478","cluster_addr":"172.21.15.117:6819/3343342478","heartbeat_back_addr":"172.21.15.117:6823/3343342478","heartbeat_front_addr":"172.21.15.117:6821/3343342478","state":["exists","up"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6803","nonce":2905616722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6807","nonce":2905616722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6816","nonce":2905616722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6811","nonce":2905616722}]},"public_addr":"172.21.15.188:6803/2905616722","cluster_addr":"172.21.15.188:6807/2905616722","heartbeat_back_addr":"172.21.15.188:6816/2905616722","heartbeat_front_addr":"172.21.15.188:6811/2905616722","state":["exists","up"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6801","nonce":2698627343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6803","nonce":2698627343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6807","nonce":2698627343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6805","nonce":2698627343}]},"public_addr":"172.21.15.37:6801/2698627343","cluster_addr":"172.21.15.37:6803/2698627343","heartbeat_back_addr":"172.21.15.37:6807/2698627343","heartbeat_front_addr":"172.21.15.37:6805/2698627343","state":["exists","up"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6825","nonce":2772028538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6827","nonce":2772028538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6831","nonce":2772028538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6829","nonce":2772028538}]},"public_addr":"172.21.15.39:6825/2772028538","cluster_addr":"172.21.15.39:6827/2772028538","heartbeat_back_addr":"172.21.15.39:6831/2772028538","heartbeat_front_addr":"172.21.15.39:6829/2772028538","state":["exists","up"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6801","nonce":3124837418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6803","nonce":3124837418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6807","nonce":3124837418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6805","nonce":3124837418}]},"public_addr":"172.21.15.117:6801/3124837418","cluster_addr":"172.21.15.117:6803/3124837418","heartbeat_back_addr":"172.21.15.117:6807/3124837418","heartbeat_front_addr":"172.21.15.117:6805/3124837418","state":["exists","up"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6806","nonce":4221150049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6810","nonce":4221150049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6815","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6817","nonce":4221150049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6813","nonce":4221150049}]},"public_addr":"172.21.15.188:6806/4221150049","cluster_addr":"172.21.15.188:6810/4221150049","heartbeat_back_addr":"172.21.15.188:6817/4221150049","heartbeat_front_addr":"172.21.15.188:6813/4221150049","state":["exists","up"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6809","nonce":3581510146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6811","nonce":3581510146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6815","nonce":3581510146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6813","nonce":3581510146}]},"public_addr":"172.21.15.37:6809/3581510146","cluster_addr":"172.21.15.37:6811/3581510146","heartbeat_back_addr":"172.21.15.37:6815/3581510146","heartbeat_front_addr":"172.21.15.37:6813/3581510146","state":["exists","up"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6801","nonce":2726119652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6803","nonce":2726119652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6807","nonce":2726119652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6805","nonce":2726119652}]},"public_addr":"172.21.15.39:6801/2726119652","cluster_addr":"172.21.15.39:6803/2726119652","heartbeat_back_addr":"172.21.15.39:6807/2726119652","heartbeat_front_addr":"172.21.15.39:6805/2726119652","state":["exists","up"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6825","nonce":737103121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6827","nonce":737103121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6831","nonce":737103121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6829","nonce":737103121}]},"public_addr":"172.21.15.117:6825/737103121","cluster_addr":"172.21.15.117:6827/737103121","heartbeat_back_addr":"172.21.15.117:6831/737103121","heartbeat_front_addr":"172.21.15.117:6829/737103121","state":["exists","up"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6823","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6824","nonce":1355328456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6828","nonce":1355328456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6833","nonce":1355328456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6831","nonce":1355328456}]},"public_addr":"172.21.15.188:6824/1355328456","cluster_addr":"172.21.15.188:6828/1355328456","heartbeat_back_addr":"172.21.15.188:6833/1355328456","heartbeat_front_addr":"172.21.15.188:6831/1355328456","state":["exists","up"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6818","nonce":3693386273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6821","nonce":3693386273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6827","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6828","nonce":3693386273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6825","nonce":3693386273}]},"public_addr":"172.21.15.37:6818/3693386273","cluster_addr":"172.21.15.37:6821/3693386273","heartbeat_back_addr":"172.21.15.37:6828/3693386273","heartbeat_front_addr":"172.21.15.37:6825/3693386273","state":["exists","up"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6817","nonce":2905392027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6819","nonce":2905392027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6823","nonce":2905392027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6821","nonce":2905392027}]},"public_addr":"172.21.15.39:6817/2905392027","cluster_addr":"172.21.15.39:6819/2905392027","heartbeat_back_addr":"172.21.15.39:6823/2905392027","heartbeat_front_addr":"172.21.15.39:6821/2905392027","state":["exists","up"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6809","nonce":1375514506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6811","nonce":1375514506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6815","nonce":1375514506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6813","nonce":1375514506}]},"public_addr":"172.21.15.117:6809/1375514506","cluster_addr":"172.21.15.117:6811/1375514506","heartbeat_back_addr":"172.21.15.117:6815/1375514506","heartbeat_front_addr":"172.21.15.117:6813/1375514506","state":["exists","up"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6819","nonce":3000701187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6821","nonce":3000701187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6829","nonce":3000701187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6825","nonce":3000701187}]},"public_addr":"172.21.15.188:6819/3000701187","cluster_addr":"172.21.15.188:6821/3000701187","heartbeat_back_addr":"172.21.15.188:6829/3000701187","heartbeat_front_addr":"172.21.15.188:6825/3000701187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.056258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.131369+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.987287+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.868903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.213442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.953995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.598108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.496609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.725374+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.710839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.990973+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.928910+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.261435+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.781696+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.550782+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.711709+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:01:09.480 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:01:09.481 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:01:09.776 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:09.776 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":30,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:01:08.729364+0000","last_up_change":"2024-09-16T04:00:57.414587+0000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:00:57.133339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:01:05.007470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6817","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6819","nonce":3422299298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6824","nonce":3422299298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6831","nonce":3422299298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6829","nonce":3422299298}]},"public_addr":"172.21.15.37:6819/3422299298","cluster_addr":"172.21.15.37:6824/3422299298","heartbeat_back_addr":"172.21.15.37:6831/3422299298","heartbeat_front_addr":"172.21.15.37:6829/3422299298","state":["exists","up"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6809","nonce":1856047451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6811","nonce":1856047451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6815","nonce":1856047451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6813","nonce":1856047451}]},"public_addr":"172.21.15.39:6809/1856047451","cluster_addr":"172.21.15.39:6811/1856047451","heartbeat_back_addr":"172.21.15.39:6815/1856047451","heartbeat_front_addr":"172.21.15.39:6813/1856047451","state":["exists","up"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6817","nonce":3343342478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6819","nonce":3343342478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6823","nonce":3343342478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6821","nonce":3343342478}]},"public_addr":"172.21.15.117:6817/3343342478","cluster_addr":"172.21.15.117:6819/3343342478","heartbeat_back_addr":"172.21.15.117:6823/3343342478","heartbeat_front_addr":"172.21.15.117:6821/3343342478","state":["exists","up"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6803","nonce":2905616722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6807","nonce":2905616722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6816","nonce":2905616722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6811","nonce":2905616722}]},"public_addr":"172.21.15.188:6803/2905616722","cluster_addr":"172.21.15.188:6807/2905616722","heartbeat_back_addr":"172.21.15.188:6816/2905616722","heartbeat_front_addr":"172.21.15.188:6811/2905616722","state":["exists","up"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6801","nonce":2698627343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6803","nonce":2698627343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6807","nonce":2698627343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6805","nonce":2698627343}]},"public_addr":"172.21.15.37:6801/2698627343","cluster_addr":"172.21.15.37:6803/2698627343","heartbeat_back_addr":"172.21.15.37:6807/2698627343","heartbeat_front_addr":"172.21.15.37:6805/2698627343","state":["exists","up"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6825","nonce":2772028538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6827","nonce":2772028538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6831","nonce":2772028538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6829","nonce":2772028538}]},"public_addr":"172.21.15.39:6825/2772028538","cluster_addr":"172.21.15.39:6827/2772028538","heartbeat_back_addr":"172.21.15.39:6831/2772028538","heartbeat_front_addr":"172.21.15.39:6829/2772028538","state":["exists","up"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6801","nonce":3124837418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6803","nonce":3124837418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6807","nonce":3124837418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6805","nonce":3124837418}]},"public_addr":"172.21.15.117:6801/3124837418","cluster_addr":"172.21.15.117:6803/3124837418","heartbeat_back_addr":"172.21.15.117:6807/3124837418","heartbeat_front_addr":"172.21.15.117:6805/3124837418","state":["exists","up"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6806","nonce":4221150049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6810","nonce":4221150049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6815","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6817","nonce":4221150049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6813","nonce":4221150049}]},"public_addr":"172.21.15.188:6806/4221150049","cluster_addr":"172.21.15.188:6810/4221150049","heartbeat_back_addr":"172.21.15.188:6817/4221150049","heartbeat_front_addr":"172.21.15.188:6813/4221150049","state":["exists","up"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6809","nonce":3581510146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6811","nonce":3581510146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6815","nonce":3581510146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6813","nonce":3581510146}]},"public_addr":"172.21.15.37:6809/3581510146","cluster_addr":"172.21.15.37:6811/3581510146","heartbeat_back_addr":"172.21.15.37:6815/3581510146","heartbeat_front_addr":"172.21.15.37:6813/3581510146","state":["exists","up"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6801","nonce":2726119652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6803","nonce":2726119652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6807","nonce":2726119652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6805","nonce":2726119652}]},"public_addr":"172.21.15.39:6801/2726119652","cluster_addr":"172.21.15.39:6803/2726119652","heartbeat_back_addr":"172.21.15.39:6807/2726119652","heartbeat_front_addr":"172.21.15.39:6805/2726119652","state":["exists","up"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6825","nonce":737103121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6827","nonce":737103121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6831","nonce":737103121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6829","nonce":737103121}]},"public_addr":"172.21.15.117:6825/737103121","cluster_addr":"172.21.15.117:6827/737103121","heartbeat_back_addr":"172.21.15.117:6831/737103121","heartbeat_front_addr":"172.21.15.117:6829/737103121","state":["exists","up"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6823","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6824","nonce":1355328456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6828","nonce":1355328456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6833","nonce":1355328456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6831","nonce":1355328456}]},"public_addr":"172.21.15.188:6824/1355328456","cluster_addr":"172.21.15.188:6828/1355328456","heartbeat_back_addr":"172.21.15.188:6833/1355328456","heartbeat_front_addr":"172.21.15.188:6831/1355328456","state":["exists","up"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6818","nonce":3693386273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6821","nonce":3693386273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6827","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6828","nonce":3693386273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6825","nonce":3693386273}]},"public_addr":"172.21.15.37:6818/3693386273","cluster_addr":"172.21.15.37:6821/3693386273","heartbeat_back_addr":"172.21.15.37:6828/3693386273","heartbeat_front_addr":"172.21.15.37:6825/3693386273","state":["exists","up"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6817","nonce":2905392027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6819","nonce":2905392027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6823","nonce":2905392027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6821","nonce":2905392027}]},"public_addr":"172.21.15.39:6817/2905392027","cluster_addr":"172.21.15.39:6819/2905392027","heartbeat_back_addr":"172.21.15.39:6823/2905392027","heartbeat_front_addr":"172.21.15.39:6821/2905392027","state":["exists","up"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6809","nonce":1375514506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6811","nonce":1375514506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6815","nonce":1375514506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6813","nonce":1375514506}]},"public_addr":"172.21.15.117:6809/1375514506","cluster_addr":"172.21.15.117:6811/1375514506","heartbeat_back_addr":"172.21.15.117:6815/1375514506","heartbeat_front_addr":"172.21.15.117:6813/1375514506","state":["exists","up"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6819","nonce":3000701187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6821","nonce":3000701187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6829","nonce":3000701187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6825","nonce":3000701187}]},"public_addr":"172.21.15.188:6819/3000701187","cluster_addr":"172.21.15.188:6821/3000701187","heartbeat_back_addr":"172.21.15.188:6829/3000701187","heartbeat_front_addr":"172.21.15.188:6825/3000701187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.056258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.131369+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.987287+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.868903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.213442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.953995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.598108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.496609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.725374+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.710839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.990973+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.928910+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:54.261435+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.781696+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.550782+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:00:53.711709+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:01:09.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:01:09.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:01:09.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:01:09.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:01:09.795 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T04:01:09.795 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T04:01:09.796 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T04:01:09.796 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T04:01:09.797 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T04:01:09.797 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T04:01:09.797 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T04:01:09.798 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T04:01:09.798 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T04:01:09.799 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T04:01:09.799 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T04:01:09.799 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T04:01:10.190 INFO:teuthology.orchestra.run.smithi037.stdout:98784247814 2024-09-16T04:01:10.191 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:01:10.237 INFO:teuthology.orchestra.run.smithi037.stdout:98784247812 2024-09-16T04:01:10.237 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:01:10.280 INFO:teuthology.orchestra.run.smithi037.stdout:98784247813 2024-09-16T04:01:10.280 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:01:10.288 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2024-09-16T04:01:10.288 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T04:01:10.291 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2024-09-16T04:01:10.291 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:01:10.302 INFO:teuthology.orchestra.run.smithi037.stdout:98784247814 2024-09-16T04:01:10.302 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:01:10.304 INFO:teuthology.orchestra.run.smithi037.stdout:94489280519 2024-09-16T04:01:10.305 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:01:10.333 INFO:teuthology.orchestra.run.smithi037.stdout:103079215111 2024-09-16T04:01:10.333 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:01:10.347 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2024-09-16T04:01:10.348 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:01:10.355 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:10.356 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T04:01:10.360 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:10.360 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:01:10.372 INFO:teuthology.orchestra.run.smithi037.stdout:94489280518 2024-09-16T04:01:10.372 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:01:10.374 INFO:teuthology.orchestra.run.smithi037.stdout:98784247814 2024-09-16T04:01:10.375 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T04:01:10.376 INFO:teuthology.orchestra.run.smithi037.stdout:94489280518 2024-09-16T04:01:10.376 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T04:01:10.410 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:10.411 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:01:10.420 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:10.420 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:01:11.100 INFO:teuthology.orchestra.run.smithi037.stdout:94489280518 2024-09-16T04:01:11.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280519 got 94489280518 for osd.4 2024-09-16T04:01:11.215 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2024-09-16T04:01:11.269 INFO:teuthology.orchestra.run.smithi037.stdout:103079215109 2024-09-16T04:01:11.274 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215111 got 103079215110 for osd.6 2024-09-16T04:01:11.283 INFO:teuthology.orchestra.run.smithi037.stdout:98784247811 2024-09-16T04:01:11.291 INFO:teuthology.orchestra.run.smithi037.stdout:103079215109 2024-09-16T04:01:11.309 INFO:teuthology.orchestra.run.smithi037.stdout:98784247813 2024-09-16T04:01:11.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215110 got 103079215109 for osd.14 2024-09-16T04:01:11.349 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.7 2024-09-16T04:01:11.352 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215110 got 103079215109 for osd.2 2024-09-16T04:01:11.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.3 2024-09-16T04:01:11.403 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2024-09-16T04:01:11.412 INFO:teuthology.orchestra.run.smithi037.stdout:98784247812 2024-09-16T04:01:11.417 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2024-09-16T04:01:11.417 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:11.443 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:11.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247812 for osd.0 2024-09-16T04:01:11.448 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2024-09-16T04:01:11.453 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.12 2024-09-16T04:01:11.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.9 2024-09-16T04:01:11.455 INFO:teuthology.orchestra.run.smithi037.stdout:98784247813 2024-09-16T04:01:11.460 INFO:teuthology.orchestra.run.smithi037.stdout:103079215109 2024-09-16T04:01:11.467 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280518 got 94489280517 for osd.1 2024-09-16T04:01:11.467 INFO:teuthology.orchestra.run.smithi037.stdout:98784247813 2024-09-16T04:01:11.469 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2024-09-16T04:01:11.476 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280518 got 94489280517 for osd.15 2024-09-16T04:01:11.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.11 2024-09-16T04:01:11.485 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.5 2024-09-16T04:01:11.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.8 2024-09-16T04:01:11.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215110 got 103079215109 for osd.10 2024-09-16T04:01:11.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.13 2024-09-16T04:01:12.147 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:01:12.275 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:01:12.320 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T04:01:12.350 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:01:12.353 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:01:12.364 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:01:12.445 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:01:12.454 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T04:01:12.455 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:01:12.468 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:01:12.477 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T04:01:12.485 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:01:12.486 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:01:12.495 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:01:12.498 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:01:12.500 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T04:01:12.522 INFO:teuthology.orchestra.run.smithi037.stdout:94489280519 2024-09-16T04:01:12.582 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280519 got 94489280519 for osd.4 2024-09-16T04:01:12.583 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:12.870 INFO:teuthology.orchestra.run.smithi037.stdout:103079215111 2024-09-16T04:01:12.944 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215111 got 103079215111 for osd.6 2024-09-16T04:01:12.945 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.079 INFO:teuthology.orchestra.run.smithi037.stdout:98784247812 2024-09-16T04:01:13.092 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2024-09-16T04:01:13.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.7 2024-09-16T04:01:13.122 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2024-09-16T04:01:13.122 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.138 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215110 got 103079215110 for osd.14 2024-09-16T04:01:13.138 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215110 got 103079215110 for osd.2 2024-09-16T04:01:13.170 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.200 INFO:teuthology.orchestra.run.smithi037.stdout:98784247814 2024-09-16T04:01:13.201 INFO:teuthology.orchestra.run.smithi037.stdout:98784247813 2024-09-16T04:01:13.247 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.3 2024-09-16T04:01:13.248 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.263 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.0 2024-09-16T04:01:13.263 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.293 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:13.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.12 2024-09-16T04:01:13.328 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.332 INFO:teuthology.orchestra.run.smithi037.stdout:98784247814 2024-09-16T04:01:13.343 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:13.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.9 2024-09-16T04:01:13.372 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.13 2024-09-16T04:01:13.373 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.411 INFO:teuthology.orchestra.run.smithi037.stdout:94489280518 2024-09-16T04:01:13.411 INFO:teuthology.orchestra.run.smithi037.stdout:94489280518 2024-09-16T04:01:13.418 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:13.421 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2024-09-16T04:01:13.440 INFO:teuthology.orchestra.run.smithi037.stdout:98784247814 2024-09-16T04:01:13.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.8 2024-09-16T04:01:13.442 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280518 got 94489280518 for osd.15 2024-09-16T04:01:13.445 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280518 got 94489280518 for osd.1 2024-09-16T04:01:13.446 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215110 got 103079215110 for osd.10 2024-09-16T04:01:13.455 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.5 2024-09-16T04:01:13.460 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.464 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2024-09-16T04:01:13.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.11 2024-09-16T04:01:13.482 DEBUG:teuthology.parallel:result is None 2024-09-16T04:01:13.482 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:01:13.483 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:13.802 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:13.802 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:13.820 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-16T04:01:13.042687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":446872,"kb_used_data":6040,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509502568,"statfs":{"total":1546188226560,"available":1545730629632,"internally_reserved":0,"allocated":6184960,"data_stored":3182758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.330679"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.988351+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:08.988351+0000","last_peered":"2024-09-16T04:01:08.988351+0000","last_clean":"2024-09-16T04:01:08.988351+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:08.988351+0000","last_undegraded":"2024-09-16T04:01:08.988351+0000","last_fullsized":"2024-09-16T04:01:08.988351+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.737438+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:08.737438+0000","last_peered":"2024-09-16T04:01:08.737438+0000","last_clean":"2024-09-16T04:01:08.737438+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:08.737438+0000","last_undegraded":"2024-09-16T04:01:08.737438+0000","last_fullsized":"2024-09-16T04:01:08.737438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.737409+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:08.737409+0000","last_peered":"2024-09-16T04:01:08.737409+0000","last_clean":"2024-09-16T04:01:08.737409+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:08.737409+0000","last_undegraded":"2024-09-16T04:01:08.737409+0000","last_fullsized":"2024-09-16T04:01:08.737409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.985956+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:08.985956+0000","last_peered":"2024-09-16T04:01:08.985956+0000","last_clean":"2024-09-16T04:01:08.985956+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:08.985956+0000","last_undegraded":"2024-09-16T04:01:08.985956+0000","last_fullsized":"2024-09-16T04:01:08.985956+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.746422+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:08.746422+0000","last_peered":"2024-09-16T04:01:08.746422+0000","last_clean":"2024-09-16T04:01:08.746422+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:08.746422+0000","last_undegraded":"2024-09-16T04:01:08.746422+0000","last_fullsized":"2024-09-16T04:01:08.746422+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987875+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:08.987875+0000","last_peered":"2024-09-16T04:01:08.987875+0000","last_clean":"2024-09-16T04:01:08.987875+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:08.987875+0000","last_undegraded":"2024-09-16T04:01:08.987875+0000","last_fullsized":"2024-09-16T04:01:08.987875+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.744719+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:08.744719+0000","last_peered":"2024-09-16T04:01:08.744719+0000","last_clean":"2024-09-16T04:01:08.744719+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:08.744719+0000","last_undegraded":"2024-09-16T04:01:08.744719+0000","last_fullsized":"2024-09-16T04:01:08.744719+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"28'1","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987327+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:08.987327+0000","last_peered":"2024-09-16T04:01:08.987327+0000","last_clean":"2024-09-16T04:01:08.987327+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:08.987327+0000","last_undegraded":"2024-09-16T04:01:08.987327+0000","last_fullsized":"2024-09-16T04:01:08.987327+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":216,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987112+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:08.987112+0000","last_peered":"2024-09-16T04:01:08.987112+0000","last_clean":"2024-09-16T04:01:08.987112+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:08.987112+0000","last_undegraded":"2024-09-16T04:01:08.987112+0000","last_fullsized":"2024-09-16T04:01:08.987112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215111,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":323584,"data_stored":132823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":851968,"data_stored":661732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":851968,"data_stored":661732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":132823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,6,8,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:13.821 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:14.103 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:14.104 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:14.121 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-16T04:01:13.042687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":446872,"kb_used_data":6040,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509502568,"statfs":{"total":1546188226560,"available":1545730629632,"internally_reserved":0,"allocated":6184960,"data_stored":3182758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.330679"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.988351+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:08.988351+0000","last_peered":"2024-09-16T04:01:08.988351+0000","last_clean":"2024-09-16T04:01:08.988351+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:08.988351+0000","last_undegraded":"2024-09-16T04:01:08.988351+0000","last_fullsized":"2024-09-16T04:01:08.988351+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.737438+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:08.737438+0000","last_peered":"2024-09-16T04:01:08.737438+0000","last_clean":"2024-09-16T04:01:08.737438+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:08.737438+0000","last_undegraded":"2024-09-16T04:01:08.737438+0000","last_fullsized":"2024-09-16T04:01:08.737438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.737409+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:08.737409+0000","last_peered":"2024-09-16T04:01:08.737409+0000","last_clean":"2024-09-16T04:01:08.737409+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:08.737409+0000","last_undegraded":"2024-09-16T04:01:08.737409+0000","last_fullsized":"2024-09-16T04:01:08.737409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.985956+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:08.985956+0000","last_peered":"2024-09-16T04:01:08.985956+0000","last_clean":"2024-09-16T04:01:08.985956+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:08.985956+0000","last_undegraded":"2024-09-16T04:01:08.985956+0000","last_fullsized":"2024-09-16T04:01:08.985956+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.746422+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:08.746422+0000","last_peered":"2024-09-16T04:01:08.746422+0000","last_clean":"2024-09-16T04:01:08.746422+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:08.746422+0000","last_undegraded":"2024-09-16T04:01:08.746422+0000","last_fullsized":"2024-09-16T04:01:08.746422+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987875+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:08.987875+0000","last_peered":"2024-09-16T04:01:08.987875+0000","last_clean":"2024-09-16T04:01:08.987875+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:08.987875+0000","last_undegraded":"2024-09-16T04:01:08.987875+0000","last_fullsized":"2024-09-16T04:01:08.987875+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.744719+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:08.744719+0000","last_peered":"2024-09-16T04:01:08.744719+0000","last_clean":"2024-09-16T04:01:08.744719+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:08.744719+0000","last_undegraded":"2024-09-16T04:01:08.744719+0000","last_fullsized":"2024-09-16T04:01:08.744719+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"28'1","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987327+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:08.987327+0000","last_peered":"2024-09-16T04:01:08.987327+0000","last_clean":"2024-09-16T04:01:08.987327+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:08.987327+0000","last_undegraded":"2024-09-16T04:01:08.987327+0000","last_fullsized":"2024-09-16T04:01:08.987327+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":216,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987112+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:08.987112+0000","last_peered":"2024-09-16T04:01:08.987112+0000","last_clean":"2024-09-16T04:01:08.987112+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:08.987112+0000","last_undegraded":"2024-09-16T04:01:08.987112+0000","last_fullsized":"2024-09-16T04:01:08.987112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215111,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":323584,"data_stored":132823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":851968,"data_stored":661732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":851968,"data_stored":661732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":132823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,6,8,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:14.122 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:01:14.122 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T04:01:14.122 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T04:01:14.122 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T04:01:14.439 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:14.439 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T04:01:14.457 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T04:01:14.457 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T04:01:14.501 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T04:01:14.501 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}} 2024-09-16T04:01:14.501 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T04:01:14.501 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T04:01:14.502 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:01:14.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:14.798 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:14.815 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-16T04:01:13.042687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":446872,"kb_used_data":6040,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509502568,"statfs":{"total":1546188226560,"available":1545730629632,"internally_reserved":0,"allocated":6184960,"data_stored":3182758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.330679"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.988351+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:08.988351+0000","last_peered":"2024-09-16T04:01:08.988351+0000","last_clean":"2024-09-16T04:01:08.988351+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:08.988351+0000","last_undegraded":"2024-09-16T04:01:08.988351+0000","last_fullsized":"2024-09-16T04:01:08.988351+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.737438+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:08.737438+0000","last_peered":"2024-09-16T04:01:08.737438+0000","last_clean":"2024-09-16T04:01:08.737438+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:08.737438+0000","last_undegraded":"2024-09-16T04:01:08.737438+0000","last_fullsized":"2024-09-16T04:01:08.737438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.737409+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:08.737409+0000","last_peered":"2024-09-16T04:01:08.737409+0000","last_clean":"2024-09-16T04:01:08.737409+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:08.737409+0000","last_undegraded":"2024-09-16T04:01:08.737409+0000","last_fullsized":"2024-09-16T04:01:08.737409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.985956+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:08.985956+0000","last_peered":"2024-09-16T04:01:08.985956+0000","last_clean":"2024-09-16T04:01:08.985956+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:08.985956+0000","last_undegraded":"2024-09-16T04:01:08.985956+0000","last_fullsized":"2024-09-16T04:01:08.985956+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.746422+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:08.746422+0000","last_peered":"2024-09-16T04:01:08.746422+0000","last_clean":"2024-09-16T04:01:08.746422+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:08.746422+0000","last_undegraded":"2024-09-16T04:01:08.746422+0000","last_fullsized":"2024-09-16T04:01:08.746422+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987875+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:08.987875+0000","last_peered":"2024-09-16T04:01:08.987875+0000","last_clean":"2024-09-16T04:01:08.987875+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:08.987875+0000","last_undegraded":"2024-09-16T04:01:08.987875+0000","last_fullsized":"2024-09-16T04:01:08.987875+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.744719+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:08.744719+0000","last_peered":"2024-09-16T04:01:08.744719+0000","last_clean":"2024-09-16T04:01:08.744719+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:08.744719+0000","last_undegraded":"2024-09-16T04:01:08.744719+0000","last_fullsized":"2024-09-16T04:01:08.744719+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"28'1","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987327+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:08.987327+0000","last_peered":"2024-09-16T04:01:08.987327+0000","last_clean":"2024-09-16T04:01:08.987327+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:08.987327+0000","last_undegraded":"2024-09-16T04:01:08.987327+0000","last_fullsized":"2024-09-16T04:01:08.987327+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":216,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987112+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:08.987112+0000","last_peered":"2024-09-16T04:01:08.987112+0000","last_clean":"2024-09-16T04:01:08.987112+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:08.987112+0000","last_undegraded":"2024-09-16T04:01:08.987112+0000","last_fullsized":"2024-09-16T04:01:08.987112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215111,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":323584,"data_stored":132823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":851968,"data_stored":661732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":851968,"data_stored":661732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":132823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,6,8,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:14.816 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:15.102 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:15.102 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:15.119 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-16T04:01:15.043340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":446872,"kb_used_data":6040,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509502568,"statfs":{"total":1546188226560,"available":1545730629632,"internally_reserved":0,"allocated":6184960,"data_stored":3182758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002522"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.988351+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:08.988351+0000","last_peered":"2024-09-16T04:01:08.988351+0000","last_clean":"2024-09-16T04:01:08.988351+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:08.988351+0000","last_undegraded":"2024-09-16T04:01:08.988351+0000","last_fullsized":"2024-09-16T04:01:08.988351+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.737438+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:08.737438+0000","last_peered":"2024-09-16T04:01:08.737438+0000","last_clean":"2024-09-16T04:01:08.737438+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:08.737438+0000","last_undegraded":"2024-09-16T04:01:08.737438+0000","last_fullsized":"2024-09-16T04:01:08.737438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.737409+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:08.737409+0000","last_peered":"2024-09-16T04:01:08.737409+0000","last_clean":"2024-09-16T04:01:08.737409+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:08.737409+0000","last_undegraded":"2024-09-16T04:01:08.737409+0000","last_fullsized":"2024-09-16T04:01:08.737409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.985956+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:08.985956+0000","last_peered":"2024-09-16T04:01:08.985956+0000","last_clean":"2024-09-16T04:01:08.985956+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:08.985956+0000","last_undegraded":"2024-09-16T04:01:08.985956+0000","last_fullsized":"2024-09-16T04:01:08.985956+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.746422+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:08.746422+0000","last_peered":"2024-09-16T04:01:08.746422+0000","last_clean":"2024-09-16T04:01:08.746422+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:08.746422+0000","last_undegraded":"2024-09-16T04:01:08.746422+0000","last_fullsized":"2024-09-16T04:01:08.746422+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987875+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:08.987875+0000","last_peered":"2024-09-16T04:01:08.987875+0000","last_clean":"2024-09-16T04:01:08.987875+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:08.987875+0000","last_undegraded":"2024-09-16T04:01:08.987875+0000","last_fullsized":"2024-09-16T04:01:08.987875+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.744719+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:08.744719+0000","last_peered":"2024-09-16T04:01:08.744719+0000","last_clean":"2024-09-16T04:01:08.744719+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:08.744719+0000","last_undegraded":"2024-09-16T04:01:08.744719+0000","last_fullsized":"2024-09-16T04:01:08.744719+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"28'1","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987327+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:08.987327+0000","last_peered":"2024-09-16T04:01:08.987327+0000","last_clean":"2024-09-16T04:01:08.987327+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:08.987327+0000","last_undegraded":"2024-09-16T04:01:08.987327+0000","last_fullsized":"2024-09-16T04:01:08.987327+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":216,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:01:08.987112+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:08.987112+0000","last_peered":"2024-09-16T04:01:08.987112+0000","last_clean":"2024-09-16T04:01:08.987112+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:08.987112+0000","last_undegraded":"2024-09-16T04:01:08.987112+0000","last_fullsized":"2024-09-16T04:01:08.987112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215111,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215111,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215112,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":323584,"data_stored":132823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":851968,"data_stored":661732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":851968,"data_stored":661732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":132823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,6,8,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,4,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":132804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:15.120 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:01:15.120 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T04:01:15.425 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 30 2024-09-16T04:01:15.425 INFO:teuthology.orchestra.run.smithi037.stdout:fsid e6b3d5c9-0596-44ca-b271-bebb6e706efc 2024-09-16T04:01:15.425 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-09-16T04:00:41.999461+0000 2024-09-16T04:01:15.425 INFO:teuthology.orchestra.run.smithi037.stdout:modified 2024-09-16T04:01:08.729364+0000 2024-09-16T04:01:15.425 INFO:teuthology.orchestra.run.smithi037.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:crush_version 5 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:full_ratio 0.95 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:backfillfull_ratio 0.9 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:nearfull_ratio 0.85 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:require_min_compat_client luminous 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:min_compat_client jewel 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:require_osd_release squid 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:stretch_mode_enabled false 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 26 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 30 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.02 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout: removed_snaps_queue [2~1] 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:max_osd 16 2024-09-16T04:01:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6817/3422299298,v1:172.21.15.37:6819/3422299298] [v2:172.21.15.37:6822/3422299298,v1:172.21.15.37:6824/3422299298] exists,up 841234ee-1773-46d0-a547-ee692a129261 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6808/1856047451,v1:172.21.15.39:6809/1856047451] [v2:172.21.15.39:6810/1856047451,v1:172.21.15.39:6811/1856047451] exists,up 7245a6e4-d25d-406d-9ed1-47ef0c8c1078 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6816/3343342478,v1:172.21.15.117:6817/3343342478] [v2:172.21.15.117:6818/3343342478,v1:172.21.15.117:6819/3343342478] exists,up 8605b852-95b1-4488-b860-a04e018a9d96 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 up in weight 1 up_from 23 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6802/2905616722,v1:172.21.15.188:6803/2905616722] [v2:172.21.15.188:6805/2905616722,v1:172.21.15.188:6807/2905616722] exists,up 9457ba25-d686-4f3b-8b67-6db4aed0af99 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 up in weight 1 up_from 22 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6800/2698627343,v1:172.21.15.37:6801/2698627343] [v2:172.21.15.37:6802/2698627343,v1:172.21.15.37:6803/2698627343] exists,up 23118253-3204-403c-b58e-c9cf18689a9f 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 up in weight 1 up_from 23 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6824/2772028538,v1:172.21.15.39:6825/2772028538] [v2:172.21.15.39:6826/2772028538,v1:172.21.15.39:6827/2772028538] exists,up 8ed21a9c-c9d4-454b-ab83-c14538a08a09 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6800/3124837418,v1:172.21.15.117:6801/3124837418] [v2:172.21.15.117:6802/3124837418,v1:172.21.15.117:6803/3124837418] exists,up fc4474a6-228a-430e-a7f6-75c86bb7b514 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6804/4221150049,v1:172.21.15.188:6806/4221150049] [v2:172.21.15.188:6808/4221150049,v1:172.21.15.188:6810/4221150049] exists,up fb4131d0-f4b4-44bf-a88f-832b545f817e 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8 up in weight 1 up_from 22 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6808/3581510146,v1:172.21.15.37:6809/3581510146] [v2:172.21.15.37:6810/3581510146,v1:172.21.15.37:6811/3581510146] exists,up 55467173-0628-4162-8d2f-fb76319de849 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6800/2726119652,v1:172.21.15.39:6801/2726119652] [v2:172.21.15.39:6802/2726119652,v1:172.21.15.39:6803/2726119652] exists,up 66db6afc-248e-4197-a30b-2bc766b6b900 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6824/737103121,v1:172.21.15.117:6825/737103121] [v2:172.21.15.117:6826/737103121,v1:172.21.15.117:6827/737103121] exists,up 54464474-a85c-4a44-8c78-ca3439d56cf4 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6823/1355328456,v1:172.21.15.188:6824/1355328456] [v2:172.21.15.188:6826/1355328456,v1:172.21.15.188:6828/1355328456] exists,up 9dabcd8b-28a8-43b2-9636-e3bb7d7a6904 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6816/3693386273,v1:172.21.15.37:6818/3693386273] [v2:172.21.15.37:6820/3693386273,v1:172.21.15.37:6821/3693386273] exists,up 43465856-0b63-41ca-b5e5-57771dbfbe5b 2024-09-16T04:01:15.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13 up in weight 1 up_from 23 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6816/2905392027,v1:172.21.15.39:6817/2905392027] [v2:172.21.15.39:6818/2905392027,v1:172.21.15.39:6819/2905392027] exists,up 7fba1ed8-6948-4b3d-afbf-236c93b9e3e5 2024-09-16T04:01:15.428 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] [v2:172.21.15.117:6810/1375514506,v1:172.21.15.117:6811/1375514506] exists,up 5073cd26-fa48-45c5-ba56-3b0c20878c2a 2024-09-16T04:01:15.428 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15 up in weight 1 up_from 22 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] [v2:172.21.15.188:6820/3000701187,v1:172.21.15.188:6821/3000701187] exists,up 52f9cbf1-4411-47e5-a8e9-442f99215ee3 2024-09-16T04:01:15.444 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6817/3422299298,v1:172.21.15.37:6819/3422299298] [v2:172.21.15.37:6822/3422299298,v1:172.21.15.37:6824/3422299298] exists,up 841234ee-1773-46d0-a547-ee692a129261', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6808/1856047451,v1:172.21.15.39:6809/1856047451] [v2:172.21.15.39:6810/1856047451,v1:172.21.15.39:6811/1856047451] exists,up 7245a6e4-d25d-406d-9ed1-47ef0c8c1078', 'osd.2 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6816/3343342478,v1:172.21.15.117:6817/3343342478] [v2:172.21.15.117:6818/3343342478,v1:172.21.15.117:6819/3343342478] exists,up 8605b852-95b1-4488-b860-a04e018a9d96', 'osd.3 up in weight 1 up_from 23 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6802/2905616722,v1:172.21.15.188:6803/2905616722] [v2:172.21.15.188:6805/2905616722,v1:172.21.15.188:6807/2905616722] exists,up 9457ba25-d686-4f3b-8b67-6db4aed0af99', 'osd.4 up in weight 1 up_from 22 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6800/2698627343,v1:172.21.15.37:6801/2698627343] [v2:172.21.15.37:6802/2698627343,v1:172.21.15.37:6803/2698627343] exists,up 23118253-3204-403c-b58e-c9cf18689a9f', 'osd.5 up in weight 1 up_from 23 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6824/2772028538,v1:172.21.15.39:6825/2772028538] [v2:172.21.15.39:6826/2772028538,v1:172.21.15.39:6827/2772028538] exists,up 8ed21a9c-c9d4-454b-ab83-c14538a08a09', 'osd.6 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6800/3124837418,v1:172.21.15.117:6801/3124837418] [v2:172.21.15.117:6802/3124837418,v1:172.21.15.117:6803/3124837418] exists,up fc4474a6-228a-430e-a7f6-75c86bb7b514', 'osd.7 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6804/4221150049,v1:172.21.15.188:6806/4221150049] [v2:172.21.15.188:6808/4221150049,v1:172.21.15.188:6810/4221150049] exists,up fb4131d0-f4b4-44bf-a88f-832b545f817e', 'osd.8 up in weight 1 up_from 22 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6808/3581510146,v1:172.21.15.37:6809/3581510146] [v2:172.21.15.37:6810/3581510146,v1:172.21.15.37:6811/3581510146] exists,up 55467173-0628-4162-8d2f-fb76319de849', 'osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6800/2726119652,v1:172.21.15.39:6801/2726119652] [v2:172.21.15.39:6802/2726119652,v1:172.21.15.39:6803/2726119652] exists,up 66db6afc-248e-4197-a30b-2bc766b6b900', 'osd.10 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6824/737103121,v1:172.21.15.117:6825/737103121] [v2:172.21.15.117:6826/737103121,v1:172.21.15.117:6827/737103121] exists,up 54464474-a85c-4a44-8c78-ca3439d56cf4', 'osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6823/1355328456,v1:172.21.15.188:6824/1355328456] [v2:172.21.15.188:6826/1355328456,v1:172.21.15.188:6828/1355328456] exists,up 9dabcd8b-28a8-43b2-9636-e3bb7d7a6904', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6816/3693386273,v1:172.21.15.37:6818/3693386273] [v2:172.21.15.37:6820/3693386273,v1:172.21.15.37:6821/3693386273] exists,up 43465856-0b63-41ca-b5e5-57771dbfbe5b', 'osd.13 up in weight 1 up_from 23 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6816/2905392027,v1:172.21.15.39:6817/2905392027] [v2:172.21.15.39:6818/2905392027,v1:172.21.15.39:6819/2905392027] exists,up 7fba1ed8-6948-4b3d-afbf-236c93b9e3e5', 'osd.14 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] [v2:172.21.15.117:6810/1375514506,v1:172.21.15.117:6811/1375514506] exists,up 5073cd26-fa48-45c5-ba56-3b0c20878c2a', 'osd.15 up in weight 1 up_from 22 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] [v2:172.21.15.188:6820/3000701187,v1:172.21.15.188:6821/3000701187] exists,up 52f9cbf1-4411-47e5-a8e9-442f99215ee3'] 2024-09-16T04:01:15.445 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T04:01:15.570 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:15.570 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "mon.a", 2024-09-16T04:01:15.570 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster": "ceph", 2024-09-16T04:01:15.570 INFO:teuthology.orchestra.run.smithi037.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T04:01:15.570 INFO:teuthology.orchestra.run.smithi037.stdout: "admin_socket_mode": "", 2024-09-16T04:01:15.570 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T04:01:15.570 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_client_required": "cephx, none", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_cluster_required": "cephx", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_debug": "true", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_service_required": "cephx", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_supported": "cephx", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_aio": "true", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_aio_reap_max": "16", 2024-09-16T04:01:15.571 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_async_discard": "false", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_block_size": "4096", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_debug_aio": "false", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_enable_discard": "false", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_flock_retry": "3", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_inject_crash": "0", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T04:01:15.572 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_ioring": "false", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_ioring_hipri": "false", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_type": "", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_allocator": "hybrid", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_buffered_io": "true", 2024-09-16T04:01:15.573 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T04:01:15.574 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T04:01:15.575 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_replay_recovery": "false", 2024-09-16T04:01:15.575 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T04:01:15.575 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T04:01:15.575 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_sync_write": "false", 2024-09-16T04:01:15.575 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T04:01:15.575 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_allocator": "hybrid", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T04:01:15.576 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_create": "true", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_db_create": "false", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_db_path": "", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_db_size": "0", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_path": "", 2024-09-16T04:01:15.577 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_size": "107374182400", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_wal_create": "false", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_wal_path": "", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bluefs": "true", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:01:15.578 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_autotune": "true", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_size": "0", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T04:01:15.579 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_type": "2q", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_clone_cow": "true", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T04:01:15.580 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_mode": "none", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_freelist": "false", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T04:01:15.581 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T04:01:15.582 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fail_eio": "false", 2024-09-16T04:01:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T04:01:15.584 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T04:01:15.585 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T04:01:15.585 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T04:01:15.585 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T04:01:15.585 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T04:01:15.588 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_blob_size": "0", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T04:01:15.589 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T04:01:15.590 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_spdk_mem": "512", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T04:01:15.591 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_tracing": "false", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T04:01:15.592 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T04:01:15.593 INFO:teuthology.orchestra.run.smithi037.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T04:01:15.593 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T04:01:15.593 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T04:01:15.593 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_cluster_require_version": "2", 2024-09-16T04:01:15.595 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_require_signatures": "false", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_require_version": "2", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_service_require_signatures": "false", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_service_require_version": "2", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_sign_messages": "true", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "chdir": "", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "client_acl_type": "", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "client_asio_thread_count": "2", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "client_cache_mid": "0.750000", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "client_cache_size": "16384", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "client_caps_release_delay": "5", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "client_check_pool_perm": "true", 2024-09-16T04:01:15.596 INFO:teuthology.orchestra.run.smithi037.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_debug_force_sync_read": "false", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_debug_getattr_caps": "false", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_debug_inject_features": "", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_die_on_failed_remount": "false", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_dirsize_rbytes": "true", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_force_lazyio": "false", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_fs": "", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_inject_release_failure": "false", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_max_inline_size": "4096", 2024-09-16T04:01:15.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mds_namespace": "", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_metadata": "", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mount_gid": "-1", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mount_timeout": "300", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mount_uid": "-1", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mountpoint": "/", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_notify_timeout": "10", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc": "true", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_max_objects": "1000", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_size": "209715200", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T04:01:15.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_permissions": "true", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_quota": "true", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_quota_df": "true", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_readahead_max_bytes": "0", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_readahead_max_periods": "4", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_readahead_min": "131072", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_reconnect_stale": "false", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_shutdown_timeout": "30", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_snapdir": ".snap", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_tick_interval": "1", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_trace": "", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_use_faked_inos": "false", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client_use_random_mds": "false", 2024-09-16T04:01:15.599 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_graylog": "false", 2024-09-16T04:01:15.600 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T04:01:15.600 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_graylog_port": "12201", 2024-09-16T04:01:15.600 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_monitors": "default=true", 2024-09-16T04:01:15.600 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_syslog": "false", 2024-09-16T04:01:15.600 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T04:01:15.600 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_syslog_level": "info", 2024-09-16T04:01:15.600 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_addr": "-", 2024-09-16T04:01:15.600 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_network": "", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_network_interface": "", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "compressor_zlib_isal": "false", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "compressor_zlib_level": "5", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "compressor_zstd_level": "1", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T04:01:15.601 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location_hook": "", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location_hook_timeout": "10", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "daemonize": "false", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "daos_pool": "tank", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T04:01:15.602 INFO:teuthology.orchestra.run.smithi037.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_alienstore": "0/5", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_asok": "1/5", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_asok_assert_abort": "false", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_auth": "20/20", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_bdev": "1/3", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_bluefs": "1/5", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_bluestore": "1/5", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_buffer": "0/1", 2024-09-16T04:01:15.603 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T04:01:15.604 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T04:01:15.604 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_cephsqlite": "0/5", 2024-09-16T04:01:15.604 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_client": "0/5", 2024-09-16T04:01:15.604 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_compressor": "1/5", 2024-09-16T04:01:15.604 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_context": "0/1", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_crush": "1/1", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_crypto": "1/5", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_cyanstore": "0/5", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_dpdk": "1/5", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_eventtrace": "1/5", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_filer": "0/1", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_filestore": "1/3", 2024-09-16T04:01:15.605 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_finisher": "1/1", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_fuse": "1/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_javaclient": "1/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_journal": "1/3", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_journaler": "0/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_kstore": "1/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_lockdep": "0/1", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mclock": "1/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds": "1/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_balancer": "1/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_locker": "1/5", 2024-09-16T04:01:15.606 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_log": "1/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_migrator": "1/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_memstore": "1/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mgr": "2/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mgrc": "1/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mon": "20/20", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_monc": "0/10", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_ms": "1/1", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_none": "0/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_objclass": "0/5", 2024-09-16T04:01:15.607 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_objectcacher": "0/5", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_objecter": "0/1", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_optracker": "0/5", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_osd": "1/5", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_paxos": "20/20", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_perfcounter": "1/5", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_prioritycache": "1/5", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rados": "0/5", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rbd": "0/5", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T04:01:15.608 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rbd_replay": "0/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_refs": "0/0", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_reserver": "1/1", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw": "1/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_access": "1/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_flight": "1/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_sync": "1/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rocksdb": "4/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore": "0/5", 2024-09-16T04:01:15.609 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_backref": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_cache": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_device": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_epm": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_journal": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_lba": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_odata": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_omap": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_onode": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_t": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_tm": "0/5", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_striper": "0/1", 2024-09-16T04:01:15.610 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_test": "0/5", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_throttle": "1/1", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_timer": "0/1", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_tp": "0/5", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_trace": "1/5", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "device_failure_prediction_mode": "none", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "err_to_graylog": "false", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "err_to_journald": "false", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "err_to_stderr": "true", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "err_to_syslog": "false", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event_tracing": "false", 2024-09-16T04:01:15.611 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_http_port": "9926", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_prio_limit": "5", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_sort_metrics": "true", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_stats_period": "5", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "fake_statfs_for_testing": "0", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "fatal_signal_handlers": "true", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "filer_max_purge_ops": "10", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "filer_max_truncate_ops": "128", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_blackhole": "false", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T04:01:15.612 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_btrfs_snap": "true", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_caller_concurrency": "10", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_debug_omap_check": "false", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_debug_verify_split": "false", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_dump_file": "", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fadvise": "true", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fail_eio": "true", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T04:01:15.613 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fd_cache_size": "128", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fiemap": "false", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_inject_stall": "0", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_journal_parallel": "false", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_journal_trailing": "false", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_journal_writeahead": "false", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_kill_at": "0", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T04:01:15.614 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_merge_threshold": "-10", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T04:01:15.615 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_odsync_write": "false", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_omap_backend_path": "", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_op_threads": "2", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_punch_hole": "false", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T04:01:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_ops": "50", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_seek_data_hole": "false", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_sloppy_crc": "false", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_splice": "false", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_split_multiple": "2", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_split_rand_factor": "20", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_update_to": "1000", 2024-09-16T04:01:15.617 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_xfs_extsize": "false", 2024-09-16T04:01:15.618 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_zfs_snap": "false", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fio_dir": "/tmp/fio", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "e6b3d5c9-0596-44ca-b271-bebb6e706efc", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_allow_other": "true", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_big_writes": "true", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_debug": "false", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_default_permissions": "false", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_disable_pagecache": "false", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_max_write": "0", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_multithreaded": "true", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_require_active_mds": "true", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_set_user_groups": "true", 2024-09-16T04:01:15.619 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_splice_move": "true", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_splice_read": "true", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_splice_write": "true", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "gss_target_name": "ceph", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "heartbeat_file": "", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "heartbeat_inject_failure": "0", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "heartbeat_interval": "5", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "host": "smithi037", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T04:01:15.620 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "inject_early_sigterm": "false", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "jaeger_agent_port": "6799", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "jaeger_tracing_enable": "false", 2024-09-16T04:01:15.621 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_aio": "true", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_align_min_size": "65536", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_block_align": "true", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_block_size": "4096", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_dio": "true", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_discard": "false", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_force_aio": "false", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_ignore_corruption": "false", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_max_write_entries": "100", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_replay_from": "0", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T04:01:15.622 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_write_header_frequency": "0", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_zero_on_create": "false", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "journaler_prefetch_periods": "10", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "journaler_prezero_periods": "5", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "journaler_write_head_interval": "15", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "key": "", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "keyfile": "", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_backend": "rocksdb", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_max_bytes": "67108864", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_max_ops": "512", 2024-09-16T04:01:15.623 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_onode_map_size": "1024", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_sync_transaction": "false", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "librados_thread_count": "2", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "lockdep": "false", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "lockdep_force_backtrace": "false", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "log_coarse_timestamps": "true", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "log_flush_on_exit": "false", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "log_graylog_port": "12201", 2024-09-16T04:01:15.624 INFO:teuthology.orchestra.run.smithi037.stdout: "log_max_new": "1000", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "log_max_recent": "10000", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "log_stderr_prefix": "", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_file": "true", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_graylog": "false", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_journald": "false", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_stderr": "false", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_syslog": "false", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_action_on_write_error": "1", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_alternate_name_max": "8192", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_asio_thread_count": "2", 2024-09-16T04:01:15.625 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_export_pin": "true", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_interval": "10", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_max": "-1", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_max_until": "-1", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_merge_size": "50", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T04:01:15.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_mode": "0", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_split_bits": "3", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_split_size": "10000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T04:01:15.627 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_mid": "0.700000", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_trim_interval": "1", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T04:01:15.628 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_debug_auth_pins": "false", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_debug_frag": "false", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_debug_scatterstat": "false", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_debug_subtrees": "false", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_default_dir_hash": "2", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_defer_session_stale": "true", 2024-09-16T04:01:15.629 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dir_max_entries": "0", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dir_prefetch": "true", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_early_reply": "true", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_enable_op_tracker": "true", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_enforce_unique_name": "true", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T04:01:15.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_health_dummy": "false", 2024-09-16T04:01:15.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_join_fs": "", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_journal_format": "1", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_create_at": "0", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_export_at": "0", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_import_at": "0", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_journal_at": "0", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T04:01:15.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_link_at": "0", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_openc_at": "0", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_rename_at": "0", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_max_events": "-1", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_max_segments": "128", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_pause": "false", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_segment_size": "0", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T04:01:15.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_trim_threshold": "128", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_completed_requests": "100000", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_export_size": "20971520", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_file_recover": "32", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_purge_files": "64", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_purge_ops": "8192", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T04:01:15.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_metrics_update_interval": "2", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_min_caps_per_client": "100", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_numa_node": "-1", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_history_duration": "600", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_history_size": "20", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_log_threshold": "5", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_ping_grace": "15", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_ping_interval": "5", 2024-09-16T04:01:15.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_max_caps": "30000", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_replay_interval": "1.000000", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_root_ino_gid": "0", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_root_ino_uid": "0", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T04:01:15.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_shutdown_check": "0", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_skip_ino": "0", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_snap_min_uid": "0", 2024-09-16T04:01:15.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_snap_rstat": "false", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_symlink_recovery": "true", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_thrash_exports": "0", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_thrash_fragments": "0", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_tick_interval": "5.000000", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_valgrind_exit": "false", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_verify_backtrace": "1", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_verify_scatter": "false", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_wipe_sessions": "false", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mempool_debug": "false", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T04:01:15.638 INFO:teuthology.orchestra.run.smithi037.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "memstore_page_set": "false", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "memstore_page_size": "65536", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_client_bytes": "134217728", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_client_messages": "512", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_disabled_modules": "", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T04:01:15.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_mds_messages": "128", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_mon_messages": "128", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_op_history_duration": "600", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_op_history_size": "20", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_op_log_threshold": "5", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_osd_messages": "8192", 2024-09-16T04:01:15.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_pool": "true", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_standby_modules": "true", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_stats_period": "5", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_stats_threshold": "5", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_tick_period": "2", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_allow_pool_delete": "true", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_bytes": "104857600", 2024-09-16T04:01:15.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_directed_command_retry": "5", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_target_rank": "-1", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_file_level": "debug", 2024-09-16T04:01:15.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_compact_on_start": "false", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_compact_on_trim": "true", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T04:01:15.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cpu_threads": "4", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_data_avail_crit": "5", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_data_avail_warn": "5", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_dump_json": "false", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_extra_checks": "false", 2024-09-16T04:01:15.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_down_mkfs_grace": "120", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_down_uptime_grace": "60", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_election_default_strategy": "1", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_election_timeout": "5.000000", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T04:01:15.645 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_enable_op_tracker": "true", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_fake_pool_delete": "false", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_force_quorum_join": "false", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_log_update_period": "5", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_max_detail": "50", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_to_clog": "true", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_host": "172.21.15.37,172.21.15.39,172.21.15.117", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_host_override": "", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_initial_members": "", 2024-09-16T04:01:15.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_lease": "5.000000", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_log_full_interval": "50", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_log_max": "10000", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_log_max_summary": "50", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_log_epochs": "500", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T04:01:15.647 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_osd": "10000", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_memory_autotune": "true", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_memory_target": "2147483648", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_digest_period": "5", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T04:01:15.648 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_complaint_time": "30", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_history_duration": "600", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_history_size": "20", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_log_threshold": "5", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T04:01:15.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_cache_size": "500", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T04:01:15.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_report_timeout": "900", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T04:01:15.651 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T04:01:15.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_reweight_max_osds": "4", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_interval": "300", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_max_keys": "100", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_timeout": "300", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_session_timeout": "300", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_smart_report_timeout": "5", 2024-09-16T04:01:15.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stretch_pool_size": "4", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_debug": "false", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_tick_interval": "5", 2024-09-16T04:01:15.654 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_older_version": "true", 2024-09-16T04:01:15.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": "", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T04:01:15.656 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_tracing_enabled": "false", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_op_threads": "3", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_cm": "false", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_device_name": "", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T04:01:15.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_sl": "3", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_type": "ib", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_reap_threshold": "5", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_before_connect": "false", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_ipv4": "true", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_ipv6": "false", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_msgr1": "true", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_msgr2": "true", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_port_max": "7568", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_port_min": "6800", 2024-09-16T04:01:15.658 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_retry_count": "3", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_retry_delay": "5", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_client": "false", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_mds": "false", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_mgr": "false", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_mon": "false", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_osd": "false", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_client_mode": "crc secure", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_cluster_type": "", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_compress_secure": "false", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T04:01:15.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_crc_data": "true", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_crc_header": "true", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_bug": "true", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_old_message": "true", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T04:01:15.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_hugepages": "", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_lro": "true", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_pmd": "", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_port_id": "0", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dump_on_send": "false", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-16T04:01:15.661 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_delay_type": "", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_network_congestion": "0", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_socket_failures": "5000", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_max_accept_failures": "4", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_max_backoff": "15.000000", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_osd_compress_mode": "none", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T04:01:15.662 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_pq_min_cost": "65536", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_public_type": "", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_service_mode": "crc secure", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_tcp_nodelay": "true", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_type": "async+posix", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "no_config_file": "false", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_inflight_ops": "1024", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T04:01:15.663 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_timeout": "10.000000", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "objectstore_blackhole": "false", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "openssl_engine_opts": "", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_max_ops": "4", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_slop": "0.020000", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T04:01:15.664 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_asio_thread_count": "2", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backfill_scan_max": "512", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backfill_scan_min": "64", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backoff_on_peering": "false", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_beacon_report_interval": "300", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_bench_duration": "30", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T04:01:15.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_blkin_trace_all": "false", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_class_update_on_start": "true", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_client_message_cap": "256", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_client_op_priority": "63", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_client_watch_timeout": "30", 2024-09-16T04:01:15.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_command_max_records": "256", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_compact_on_start": "false", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_crush_update_on_start": "true", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T04:01:15.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_op_order": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_shutdown": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T04:01:15.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_trim_objects": "false", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T04:01:15.669 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_default_notify_timeout": "30", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_enable_op_tracker": "true", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_info": "true", 2024-09-16T04:01:15.670 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_shutdown": "true", 2024-09-16T04:01:15.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T04:01:15.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T04:01:15.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T04:01:15.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T04:01:15.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T04:01:15.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_function_tracing": "false", 2024-09-16T04:01:15.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_grace": "20", 2024-09-16T04:01:15.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_interval": "6", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_stale": "600", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T04:01:15.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_journal_size": "5120", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_kill_backfill_at": "0", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_cache_size": "50", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_dedup": "true", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_message_max": "40", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_attr_name_len": "100", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_attr_size": "0", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_backfills": "1", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_markdown_count": "5", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_markdown_period": "600", 2024-09-16T04:01:15.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_object_name_len": "2048", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_object_size": "134217728", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_pgls": "1024", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_push_cost": "8388608", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_push_objects": "10", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_scrubs": "3", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T04:01:15.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_write_size": "90", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_profile": "balanced", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T04:01:15.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_base": "805306368", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_target": "4294967296", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_target_autotune": "false", 2024-09-16T04:01:15.676 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_min_recovery_priority": "0", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_report_interval": "5", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_num_cache_shards": "32", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_numa_node": "-1", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-16T04:01:15.677 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_objecter_finishers": "1", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_objectstore": "bluestore", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_objectstore_fuse": "false", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_objectstore_tracing": "false", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_history_duration": "600", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_history_size": "20", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_log_threshold": "5", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_shards": "0", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T04:01:15.678 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_thread_timeout": "15", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_open_classes_on_start": "true", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_os_flags": "0", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_peering_op_priority": "255", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_delete_priority": "5", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T04:01:15.679 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_crimson": "false", 2024-09-16T04:01:15.680 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flags": "0", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_min_size": "0", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T04:01:15.681 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_size": "2", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_type": "replicated", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_cost": "20971520", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_active": "0", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T04:01:15.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_op_priority": "3", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_priority": "5", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_repair_during_recovery": "false", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T04:01:15.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_cost": "52428800", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_end_hour": "0", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T04:01:15.684 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_priority": "5", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_skip_data_digest": "false", 2024-09-16T04:01:15.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_smart_report_timeout": "5", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_priority": "5", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_target_transaction_size": "30", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T04:01:15.686 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tracing": "false", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_use_stale_snap": "false", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_kill_at": "0", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_max_join_drift": "10", 2024-09-16T04:01:15.687 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_min": "500", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_min_wait": "0.050000", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_service_trim_max": "500", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_service_trim_min": "250", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_stash_full_interval": "25", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_trim_max": "500", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_trim_min": "250", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "perf": "true", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "-", 2024-09-16T04:01:15.688 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrv": "", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "public_bind_addr": "-", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "public_network": "", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "public_network_interface": "", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "qat_compressor_enabled": "false", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rados_mon_op_timeout": "0", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rados_osd_op_timeout": "0", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rados_tracing": "false", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_atime_update_interval": "60", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T04:01:15.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache": "true", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_policy": "writearound", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_size": "33554432", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_compression_hint": "none", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T04:01:15.690 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_clone_format": "auto", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_data_pool": "", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_features": "61", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_format": "2", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_map_options": "", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_order": "22", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_pool": "rbd", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_stripe_count": "0", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T04:01:15.691 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_io_scheduler": "simple", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_order": "24", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_pool": "", 2024-09-16T04:01:15.692 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_splay_width": "4", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T04:01:15.693 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T04:01:15.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T04:01:15.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_op_threads": "1", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_plugins": "", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T04:01:15.696 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T04:01:15.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_tracing": "false", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_validate_names": "true", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_validate_pool": "true", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "restapi_base_url": "", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "restapi_log_level": "", 2024-09-16T04:01:15.698 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_admin_entry": "admin", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_backend_store": "rados", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_barbican_url": "", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_beast_enable_async": "true", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T04:01:15.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cache_enabled": "true", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_config_store": "rados", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_content_length_compat": "false", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cross_domain_policy": "", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T04:01:15.700 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T04:01:15.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T04:01:15.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T04:01:15.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d4n_port": "6379", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_log_window": "30", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T04:01:15.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T04:01:15.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dns_name": "", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dns_s3website_name": "", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T04:01:15.706 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_ops_log": "false", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_static_website": "false", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_usage_log": "false", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_expose_bucket": "false", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_extended_http_attrs": "", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_filter": "none", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_objs": "32", 2024-09-16T04:01:15.707 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_init_timeout": "300", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T04:01:15.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_password": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_project": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_token": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_user": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_api_version": "2", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T04:01:15.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_url": "", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_max_objs": "32", 2024-09-16T04:01:15.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_max_worker": "3", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_uri": "", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_log_http_headers": "", 2024-09-16T04:01:15.711 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_attr_size": "0", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_listing_results": "1000", 2024-09-16T04:01:15.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_notify_retries": "10", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T04:01:15.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T04:01:15.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_num_control_oids": "8", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_numa_node": "-1", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_op_tracing": "false", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_opa_token": "", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_opa_url": "", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T04:01:15.715 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ops_log_rados": "false", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T04:01:15.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_print_continue": "true", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_rados_tracing": "false", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_realm": "", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_realm_id": "", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_region": "", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T04:01:15.717 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_request_uri": "", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_resolve_cname": "false", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_run_sync_thread": "true", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T04:01:15.718 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_script_uri": "", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_service_provider_name": "", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_client_id": "", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_client_secret": "", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_entry": "sts", 2024-09-16T04:01:15.719 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_key": "sts", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_auth_url": "", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_custom_header": "", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_need_stats": "true", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_tenant_name": "", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_url": "", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T04:01:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_lease_period": "120", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_thread_pool_size": "512", 2024-09-16T04:01:15.721 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_comment": "", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_createby": "", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_encoding": "", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_flag": "false", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_origin": "", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_tracker": "", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T04:01:15.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_usage_max_shards": "32", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_use_opa_authz": "false", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_counters_cache": "false", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_unique_email": "true", 2024-09-16T04:01:15.723 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_verify_ssl": "true", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zone": "", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zone_id": "", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zonegroup": "", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zonegroup_id": "", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_block_size": "4096", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T04:01:15.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T04:01:15.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_partition_filters": "false", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_perf": "false", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "run_dir": "/var/run/ceph", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_block_create": "true", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_device_size": "53687091200", 2024-09-16T04:01:15.726 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_full_integrity_check": "false", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_main_device_type": "SSD", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_segment_size": "67108864", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "set_keepcaps": "false", 2024-09-16T04:01:15.727 INFO:teuthology.orchestra.run.smithi037.stdout: "setgroup": "", 2024-09-16T04:01:15.728 INFO:teuthology.orchestra.run.smithi037.stdout: "setuser": "", 2024-09-16T04:01:15.728 INFO:teuthology.orchestra.run.smithi037.stdout: "setuser_match_path": "", 2024-09-16T04:01:15.728 INFO:teuthology.orchestra.run.smithi037.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T04:01:15.728 INFO:teuthology.orchestra.run.smithi037.stdout: "thp": "false", 2024-09-16T04:01:15.728 INFO:teuthology.orchestra.run.smithi037.stdout: "threadpool_default_timeout": "60", 2024-09-16T04:01:15.728 INFO:teuthology.orchestra.run.smithi037.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T04:01:15.728 INFO:teuthology.orchestra.run.smithi037.stdout: "throttler_perf_counter": "true" 2024-09-16T04:01:15.728 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:15.730 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T04:01:15.874 INFO:teuthology.orchestra.run.smithi037.stdout:mon.a: {} 2024-09-16T04:01:15.874 INFO:teuthology.orchestra.run.smithi037.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:01:15.887 INFO:teuthology.orchestra.run.smithi037.stdout:mon.b: {} 2024-09-16T04:01:15.887 INFO:teuthology.orchestra.run.smithi037.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:01:15.901 INFO:teuthology.orchestra.run.smithi037.stdout:mon.c: {} 2024-09-16T04:01:15.901 INFO:teuthology.orchestra.run.smithi037.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:01:15.919 DEBUG:teuthology.orchestra.run.smithi037:> type ceph-objectstore-tool 2024-09-16T04:01:15.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:01:15.923 DEBUG:teuthology.orchestra.run.smithi117:> type ceph-objectstore-tool 2024-09-16T04:01:15.929 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:01:15.929 DEBUG:teuthology.orchestra.run.smithi039:> type ceph-objectstore-tool 2024-09-16T04:01:15.935 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:01:15.935 DEBUG:teuthology.orchestra.run.smithi188:> type ceph-objectstore-tool 2024-09-16T04:01:15.941 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:01:15.942 INFO:teuthology.run_tasks:Running task rados... 2024-09-16T04:01:15.952 INFO:tasks.rados:Beginning rados... 2024-09-16T04:01:15.952 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-16T04:01:15.959 INFO:tasks.rados:joining rados 2024-09-16T04:01:15.960 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T04:01:15.960 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:01:15.960 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T04:01:15.960 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T04:01:15.961 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T04:01:15.961 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:15.961 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T04:01:15.961 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:01:15.962 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T04:01:15.962 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:01:15.962 INFO:tasks.rados:clients are ['client.0'] 2024-09-16T04:01:15.962 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-09-16T04:01:15.977 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:15.974+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:16.063 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:16.058+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:16.083 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:16.083 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:16.083 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:16.083 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:16.098 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:01:16.164 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:16.158+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:16.213 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:16.213 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:16.213 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:16.214 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:16.214 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:16.226 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:01:16.265 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:16.258+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:16.308 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:16.308 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:16.328 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:16.329 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:16.338 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:16.338 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:16.349 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:16.351 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:01:16.372 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:16.373 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:16.373 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:16.369+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:16.391 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:16.391 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:16.413 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:16.435 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:16.435 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:16.457 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:16.457 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:16.459 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:16.459 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:16.466 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:16.461+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:16.472 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:01:16.475 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:16.475 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:16.496 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:16.518 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:16.519 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:16.539 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:16.539 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:16.554 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:16.554 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:16.567 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:16.561+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:16.574 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:16.574 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:16.589 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:16.590 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:16.594 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:16.595 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:16.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:16.589+0000 7eff6d82f640 0 -- 172.21.15.37:0/1969096560 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7eff68224070 msgr2=0x7eff68226460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T04:01:16.603 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:01:16.617 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:16.617 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:16.668 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:16.667+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:16.706 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-09-16T04:01:16.707 INFO:teuthology.orchestra.run.smithi037.stderr:load: shec 2024-09-16T04:01:16.708 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T04:01:16.731 INFO:tasks.rados:starting run 0 out of 1 2024-09-16T04:01:16.731 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-16T04:01:16.731 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-09-16T04:01:16.732 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T04:01:16.750 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:16.750 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:16.765 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:01:16.814 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:16.809+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:16.869 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:16.867+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:16.935 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:16.935 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:16.949 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:01:16.970 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:16.966+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:17.057 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:17.057 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:17.070 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:01:17.072 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:17.066+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:17.173 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:17.166+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:17.189 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:17.189 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:17.202 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:01:17.274 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:17.271+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:17.327 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:17.327 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:17.340 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:01:17.376 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:17.370+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:17.460 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:17.460 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:17.474 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:01:17.477 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:17.469+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:17.577 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:17.575+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:17.596 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:17.596 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:17.609 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:01:17.678 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:17.675+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:17.746 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:17.747 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:17.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:01:17.779 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:17.775+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:17.792 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T04:01:17.813 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T04:01:17.880 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:17.874+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:17.881 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:17.881 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:17.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:01:17.985 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:17.982+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:18.069 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:18.069 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:18.081 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:18.077+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:18.083 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:01:18.182 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:18.177+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:18.256 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:18.257 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:18.271 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:01:18.283 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:18.279+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:18.384 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:18.383+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:18.398 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:18.398 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:18.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:01:18.485 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:18.482+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:18.540 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:18.540 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:18.554 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:01:18.587 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:18.583+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:18.633 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:01:18.683 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:18.683 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:18.688 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:18.683+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:18.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:01:18.740 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:01:18.788 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:18.782+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:18.792 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:18.793 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:18.802 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'unique_pool_0' created 2024-09-16T04:01:18.802 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T04:01:18.821 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:18.833 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T04:01:18.834 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:18.834 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:18.835 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T04:01:18.849 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:01:18.850 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:18.850 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:18.887 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:18.887 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:18.889 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:18.882+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:18.913 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:18.914 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:18.946 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:18.966 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:18.967 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:18.990 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:18.990 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:18.992 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:18.986+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:19.006 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:19.006 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:19.016 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:19.016 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:19.026 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:19.032 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:01:19.046 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:19.046 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:19.067 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:19.083 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:19.094 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:19.091+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:19.101 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:19.101 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:19.097+0000 7f877f7fe640 0 -- 172.21.15.37:0/2409439160 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7f876c0725b0 msgr2=0x7f876c0749a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:19.161 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:19.162 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:19.162 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:19.162 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:19.176 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:01:19.192 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:19.191+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:19.292 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:19.291+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:19.304 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:19.304 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:19.305 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:19.305 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:19.305 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:19.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:01:19.324 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:19.325 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:19.359 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:19.360 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:19.393 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:19.390+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:19.450 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.985362578, 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.56051925899999999, 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.451 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460473+0000", 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.097e-06 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460488+0000", 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5335000000000001e-05 2024-09-16T04:01:19.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460512+0000", 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4108999999999999e-05 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460533+0000", 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0721999999999999e-05 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460655+0000", 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122328 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460689+0000", 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014402+0000", 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.55371241699999996 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014725+0000", 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000323345 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.015144+0000", 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041881500000000001 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020086+0000", 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049418869999999998 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020836+0000", 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074984300000000004 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020926+0000", 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0070999999999999e-05 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020990+0000", 2024-09-16T04:01:19.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4256000000000003e-05 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412334+0000", 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.033499917, 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015286879999999999, 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.458 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 113 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412334+0000", 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412334+0000", 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412334+0000", 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.23e-07 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412341+0000", 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1199999999999996e-06 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412354+0000", 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2935999999999999e-05 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412364+0000", 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7459999999999995e-06 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.421535+0000", 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0091712579999999998 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.421664+0000", 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012879100000000001 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.421856+0000", 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019163399999999999 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425449+0000", 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035928940000000001 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427478+0000", 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0020294750000000002 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427558+0000", 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9526999999999999e-05 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427620+0000", 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2675999999999994e-05 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.463 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.033455483000001, 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016442578999999999, 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.464 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 114 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412379+0000", 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.06e-07 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412387+0000", 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1799999999999996e-06 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412397+0000", 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0236e-05 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412404+0000", 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2989999999999999e-06 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.422787+0000", 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010382561 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.422907+0000", 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120316 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.423117+0000", 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020991200000000001 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427100+0000", 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039832779999999998 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428687+0000", 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00158661 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428763+0000", 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5679000000000005e-05 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428821+0000", 2024-09-16T04:01:19.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7802e-05 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.033394486999999, 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01663504, 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.470 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 115 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:19.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412440+0000", 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.083e-06 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412538+0000", 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7979999999999994e-05 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412552+0000", 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3736e-05 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412560+0000", 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2630000000000003e-06 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424204+0000", 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011644046 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424328+0000", 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124282 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424537+0000", 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020823900000000001 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427111+0000", 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025745849999999999 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428934+0000", 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018232680000000001 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429015+0000", 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1034999999999999e-05 2024-09-16T04:01:19.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429074+0000", 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8523000000000002e-05 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.033141107999999, 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016628581, 2024-09-16T04:01:19.476 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 116 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412693+0000", 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.077e-06 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412701+0000", 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8390000000000007e-06 2024-09-16T04:01:19.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412715+0000", 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3482999999999999e-05 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412726+0000", 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0798999999999999e-05 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425538+0000", 2024-09-16T04:01:19.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012812235999999999 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425665+0000", 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000126995 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425861+0000", 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019642699999999999 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427642+0000", 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001781244 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429160+0000", 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001517795 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429264+0000", 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010388299999999999 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429321+0000", 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6802999999999998e-05 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.033093601000001, 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019211259000000001, 2024-09-16T04:01:19.482 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 117 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412741+0000", 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8700000000000005e-07 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412747+0000", 2024-09-16T04:01:19.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1929999999999998e-06 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412757+0000", 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0039e-05 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412764+0000", 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3259999999999998e-06 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426386+0000", 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013621668999999999 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426547+0000", 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000161192 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426753+0000", 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020624599999999999 2024-09-16T04:01:19.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428173+0000", 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001419305 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431799+0000", 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036262320000000001 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431890+0000", 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0987000000000001e-05 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431951+0000", 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1383000000000005e-05 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765519+0000", 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.680314439, 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013028045, 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 159 2024-09-16T04:01:19.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765519+0000", 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765519+0000", 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765520+0000", 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2139999999999999e-06 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765528+0000", 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5660000000000001e-06 2024-09-16T04:01:19.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765542+0000", 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3856000000000001e-05 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765550+0000", 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8519999999999993e-06 2024-09-16T04:01:19.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.772952+0000", 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0074015000000000001 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.773177+0000", 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022507500000000001 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.773465+0000", 2024-09-16T04:01:19.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000288254 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778125+0000", 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046601209999999997 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778278+0000", 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000152301 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778438+0000", 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016031899999999999 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778547+0000", 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010898700000000001 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.495 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.680267877999999, 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016729104000000002, 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 160 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765567+0000", 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765573+0000", 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0260000000000004e-06 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765583+0000", 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765590+0000", 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4800000000000004e-06 2024-09-16T04:01:19.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774469+0000", 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.008879184 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774666+0000", 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019695699999999999 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774998+0000", 2024-09-16T04:01:19.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00033151099999999999 2024-09-16T04:01:19.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778132+0000", 2024-09-16T04:01:19.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031338849999999999 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782066+0000", 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039342630000000003 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782181+0000", 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000115093 2024-09-16T04:01:19.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782295+0000", 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113726 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.680231134, 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017211085000000001, 2024-09-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 161 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765603+0000", 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765727+0000", 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124184 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765750+0000", 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2296000000000001e-05 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765759+0000", 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2760000000000006e-06 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775766+0000", 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010007085000000001 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775856+0000", 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9683000000000002e-05 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.776003+0000", 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014695999999999999 2024-09-16T04:01:19.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779325+0000", 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033221930000000002 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782596+0000", 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032713519999999999 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782726+0000", 2024-09-16T04:01:19.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013011500000000001 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782813+0000", 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7174000000000003e-05 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.680056732000001, 2024-09-16T04:01:19.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017217072999999999, 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 162 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:19.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765778+0000", 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765784+0000", 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2659999999999998e-06 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765794+0000", 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.029e-05 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765809+0000", 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4474999999999999e-05 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777438+0000", 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011628929 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777529+0000", 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1577999999999996e-05 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777666+0000", 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013660699999999999 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781937+0000", 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042713660000000004 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782885+0000", 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00094829699999999996 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782947+0000", 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1261999999999997e-05 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782994+0000", 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7139999999999999e-05 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.514 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.680006348999999, 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01977027, 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.515 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 163 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765828+0000", 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765834+0000", 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7830000000000004e-06 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765844+0000", 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0149000000000001e-05 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765852+0000", 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0199999999999994e-06 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778628+0000", 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012776183999999999 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778715+0000", 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6941000000000007e-05 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778850+0000", 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013517800000000001 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783007+0000", 2024-09-16T04:01:19.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041571960000000002 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785458+0000", 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002450626 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785539+0000", 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1213999999999999e-05 2024-09-16T04:01:19.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785597+0000", 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8258999999999999e-05 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.679960672, 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019887134000000001, 2024-09-16T04:01:19.521 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 164 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765874+0000", 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0130000000000001e-06 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765996+0000", 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122039 2024-09-16T04:01:19.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766015+0000", 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9405000000000001e-05 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766025+0000", 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0096e-05 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779427+0000", 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013401569 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779649+0000", 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022229500000000001 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.780098+0000", 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00044856699999999998 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783018+0000", 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002919792 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785656+0000", 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026383489999999999 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785714+0000", 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8177000000000002e-05 2024-09-16T04:01:19.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785760+0000", 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5831999999999998e-05 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.679789750000001, 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021760347999999999, 2024-09-16T04:01:19.527 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 165 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766045+0000", 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766051+0000", 2024-09-16T04:01:19.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8140000000000003e-06 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766062+0000", 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0592e-05 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766069+0000", 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3629999999999996e-06 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781249+0000", 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015179349999999999 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781356+0000", 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000107747 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781496+0000", 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013985500000000001 2024-09-16T04:01:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783295+0000", 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017986199999999999 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787679+0000", 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0043837559999999999 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787752+0000", 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3192e-05 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787804+0000", 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2182999999999998e-05 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.934594+0000", 2024-09-16T04:01:19.533 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.511239213, 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013424388000000001, 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 184 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934594+0000", 2024-09-16T04:01:19.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934594+0000", 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934595+0000", 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934601+0000", 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0399999999999998e-06 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934611+0000", 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0438999999999999e-05 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934619+0000", 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5769999999999996e-06 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.942835+0000", 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082160110000000005 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.942952+0000", 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000117403 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.943145+0000", 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019284400000000001 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946721+0000", 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035755409999999998 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.947875+0000", 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011540260000000001 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.947952+0000", 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6867999999999997e-05 2024-09-16T04:01:19.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948019+0000", 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6905999999999999e-05 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.934710+0000", 2024-09-16T04:01:19.539 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.511123384000001, 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013587647, 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 185 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934710+0000", 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934710+0000", 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934712+0000", 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.697e-06 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934903+0000", 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000191447 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934942+0000", 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8590999999999997e-05 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934962+0000", 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0324999999999999e-05 2024-09-16T04:01:19.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.943925+0000", 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0089627860000000004 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.944057+0000", 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000132287 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.944272+0000", 2024-09-16T04:01:19.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021434199999999999 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948036+0000", 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037647499999999999 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948107+0000", 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0884999999999996e-05 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948187+0000", 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9961000000000002e-05 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948298+0000", 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011057600000000001 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.545 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935003+0000", 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.510830649999999, 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015768138000000001, 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.546 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 186 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935003+0000", 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935003+0000", 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935005+0000", 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7099999999999999e-06 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935017+0000", 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2836e-05 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935044+0000", 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6358000000000001e-05 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935062+0000", 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8515000000000001e-05 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.945879+0000", 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010816345 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.945998+0000", 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000119162 2024-09-16T04:01:19.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946187+0000", 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000189261 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949633+0000", 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00344641 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950612+0000", 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00097849500000000002 2024-09-16T04:01:19.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950707+0000", 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4846999999999995e-05 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950771+0000", 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4199e-05 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.510744153000001, 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015994999999999999, 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.552 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 187 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935091+0000", 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.567e-06 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935103+0000", 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1851999999999999e-05 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935127+0000", 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4596e-05 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935142+0000", 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4986e-05 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946804+0000", 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011661985 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946926+0000", 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000121455 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.947115+0000", 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018956100000000001 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950511+0000", 2024-09-16T04:01:19.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033955729999999998 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950887+0000", 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037619500000000001 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951025+0000", 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013776399999999999 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951084+0000", 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9466000000000003e-05 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.510666620999999, 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016322997999999998, 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.558 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 188 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:19.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935168+0000", 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4270000000000001e-06 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935319+0000", 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000150383 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935346+0000", 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7750999999999999e-05 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935392+0000", 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5361000000000002e-05 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948392+0000", 2024-09-16T04:01:19.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013000429000000001 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948504+0000", 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011150900000000001 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948716+0000", 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021198300000000001 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951106+0000", 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0023904899999999999 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951365+0000", 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025906300000000002 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951438+0000", 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2571999999999996e-05 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951490+0000", 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2030000000000002e-05 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.510407008000001, 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016713338000000001, 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.564 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 189 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:19.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935428+0000", 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935443+0000", 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4599e-05 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935468+0000", 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4788000000000001e-05 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935486+0000", 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8249000000000001e-05 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949701+0000", 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014214839 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949818+0000", 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011776499999999999 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950001+0000", 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018260100000000001 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951914+0000", 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001913049 2024-09-16T04:01:19.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952007+0000", 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3006999999999999e-05 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952083+0000", 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5722000000000004e-05 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952140+0000", 2024-09-16T04:01:19.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7006000000000003e-05 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:207 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:01.813887+0000", 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.631946250999999, 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012266131, 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:19.570 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 207 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.813887+0000", 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.813887+0000", 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.813888+0000", 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4e-07 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.813894+0000", 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2060000000000004e-06 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.813905+0000", 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0329000000000001e-05 2024-09-16T04:01:19.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.813923+0000", 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8022000000000001e-05 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.822380+0000", 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0084575480000000005 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.822611+0000", 2024-09-16T04:01:19.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000231225 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.822938+0000", 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000326223 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.825864+0000", 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029264690000000001 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.826016+0000", 2024-09-16T04:01:19.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015207400000000001 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.826095+0000", 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9258000000000006e-05 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:01.826153+0000", 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7936999999999998e-05 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.575 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:19.576 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:19.576 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:19.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:01:19.578 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:19.570+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:19.678 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:19.673+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:19.709 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:19.710 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:19.710 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:19.710 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:19.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:01:19.779 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:19.774+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:19.814 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-09-16T04:01:19.815 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T04:01:19.815 INFO:teuthology.orchestra.run.smithi037.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T04:01:19.852 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T04:01:19.853 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:19.853 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:19.854 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:19.854 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:19.854 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:19.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:01:19.880 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:19.874+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:19.981 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:19.974+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:19.987 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:19.987 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:19.987 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:19.987 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:01:19.987 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.988 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:54 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'32, mlcod=25'29)", 2024-09-16T04:01:19.988 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:19.988 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.853975593000001, 2024-09-16T04:01:19.988 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005928254, 2024-09-16T04:01:19.988 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:19.988 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:19.989 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.989 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:19.989 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.989 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:19.989 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:19.989 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:19.989 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.990 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.990 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:19.990 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:19.990 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:19.990 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.991 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.991 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:19.991 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128770+0000", 2024-09-16T04:01:19.991 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:01:19.991 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.991 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.991 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:19.991 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128784+0000", 2024-09-16T04:01:19.992 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3959e-05 2024-09-16T04:01:19.992 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.992 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.992 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:19.993 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128819+0000", 2024-09-16T04:01:19.993 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5237999999999997e-05 2024-09-16T04:01:19.993 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.993 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.994 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:19.994 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128838+0000", 2024-09-16T04:01:19.994 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8754999999999999e-05 2024-09-16T04:01:19.994 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.994 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.995 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:19.995 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128996+0000", 2024-09-16T04:01:19.995 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000158419 2024-09-16T04:01:19.995 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.995 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.995 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:19.996 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129033+0000", 2024-09-16T04:01:19.996 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6905e-05 2024-09-16T04:01:19.996 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.996 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.997 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:19.997 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.134608+0000", 2024-09-16T04:01:19.997 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055745869999999998 2024-09-16T04:01:19.997 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.997 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.134697+0000", 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8701000000000003e-05 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:55 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'33, mlcod=25'29)", 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.129935+0000", 2024-09-16T04:01:19.998 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.852809242999999, 2024-09-16T04:01:19.999 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005436906, 2024-09-16T04:01:19.999 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:19.999 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:19.999 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:19.999 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:19.999 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:19.999 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:19.999 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129935+0000", 2024-09-16T04:01:20.000 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.000 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.000 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.000 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.000 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129935+0000", 2024-09-16T04:01:20.000 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.000 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.000 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129936+0000", 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.556e-06 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130009+0000", 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.2582000000000004e-05 2024-09-16T04:01:20.001 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130033+0000", 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4032e-05 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130046+0000", 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2699e-05 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.002 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130161+0000", 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00011516899999999999 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130194+0000", 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2795e-05 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.003 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.135305+0000", 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051117280000000003 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.135372+0000", 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6345000000000003e-05 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.004 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:59 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'37, mlcod=25'33)", 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.140294+0000", 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.84244954, 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0049622499999999996, 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.140294+0000", 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.005 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.140294+0000", 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.140296+0000", 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.006 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.140309+0000", 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2839e-05 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.140339+0000", 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9838000000000001e-05 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.140354+0000", 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5356e-05 2024-09-16T04:01:20.007 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.140469+0000", 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000114979 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.140506+0000", 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6974000000000002e-05 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.008 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.145160+0000", 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0046539340000000002 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.145257+0000", 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6733999999999999e-05 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:87 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'57, mlcod=25'51)", 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.784491883000001, 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056693940000000003, 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.010 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198253+0000", 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2699999999999999e-06 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198371+0000", 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000117349 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.011 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198394+0000", 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3768e-05 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198411+0000", 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6186000000000001e-05 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198538+0000", 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000127842 2024-09-16T04:01:20.012 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198574+0000", 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5451000000000003e-05 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.203863+0000", 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052887919999999996 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.013 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.203921+0000", 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8736000000000001e-05 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:113 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'75, mlcod=25'70)", 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.421954+0000", 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.560790181000002, 2024-09-16T04:01:20.014 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051301749999999998, 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.421954+0000", 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.421954+0000", 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.015 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.421955+0000", 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.333e-06 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.421966+0000", 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0756e-05 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.016 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.421986+0000", 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9822000000000001e-05 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.421998+0000", 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2102e-05 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.422118+0000", 2024-09-16T04:01:20.017 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000120679 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.422161+0000", 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.2926999999999997e-05 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.427008+0000", 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0048465039999999997 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.018 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.019 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.427084+0000", 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6051999999999996e-05 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:114 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'76, mlcod=25'71)", 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.423158+0000", 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.559585831, 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0050553280000000004, 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.020 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.423158+0000", 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.423158+0000", 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.021 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.423159+0000", 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3200000000000001e-06 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.423168+0000", 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.816e-06 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.423186+0000", 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7431999999999998e-05 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.022 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.423197+0000", 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1833e-05 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.423367+0000", 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000169906 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.423424+0000", 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6356e-05 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.023 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.428152+0000", 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0047281789999999999 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.428213+0000", 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1486000000000001e-05 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.024 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:120 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'82, mlcod=25'78)", 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.435845+0000", 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.546898934000001, 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052667130000000001, 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435845+0000", 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.025 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435845+0000", 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435847+0000", 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6530000000000001e-06 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435928+0000", 2024-09-16T04:01:20.026 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1687000000000006e-05 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435957+0000", 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8694999999999999e-05 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435970+0000", 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3315999999999999e-05 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436070+0000", 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.9270000000000003e-05 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436106+0000", 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6021e-05 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.441056+0000", 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0049504750000000002 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.028 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.441112+0000", 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5596000000000001e-05 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:121 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'83, mlcod=25'78)", 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.545852286999999, 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0070855379999999997, 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.029 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.030 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436894+0000", 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8700000000000001e-06 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436963+0000", 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9152000000000005e-05 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436991+0000", 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8388e-05 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.031 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.437006+0000", 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4883e-05 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.438211+0000", 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0012053599999999999 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.032 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.438267+0000", 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5810000000000003e-05 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.443897+0000", 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056298700000000004 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.443977+0000", 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0204999999999995e-05 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.033 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:129 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'87, mlcod=25'85)", 2024-09-16T04:01:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.389499175000001, 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054743200000000004, 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.036 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593246+0000", 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6759999999999999e-06 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593258+0000", 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1715999999999999e-05 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593286+0000", 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7912e-05 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.037 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593299+0000", 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3352e-05 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593426+0000", 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000127002 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593482+0000", 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5952999999999998e-05 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.038 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598617+0000", 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051345549999999998 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598719+0000", 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000102154 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.039 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:130 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'88, mlcod=25'85)", 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.593454+0000", 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.389289922, 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053433049999999996, 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593454+0000", 2024-09-16T04:01:20.040 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593454+0000", 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593456+0000", 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6130000000000001e-06 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.041 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593466+0000", 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0686000000000001e-05 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593486+0000", 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9542e-05 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593500+0000", 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3776e-05 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.042 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594031+0000", 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00053142100000000002 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594069+0000", 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8244000000000002e-05 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598727+0000", 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0046581239999999996 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.043 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598797+0000", 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9899000000000003e-05 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:131 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'89, mlcod=25'85)", 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.388076589000001, 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058446720000000004, 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.044 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.045 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594669+0000", 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0700000000000001e-06 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594744+0000", 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4637999999999999e-05 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.046 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594769+0000", 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5245e-05 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594782+0000", 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2840000000000001e-05 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594871+0000", 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.9091000000000005e-05 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.047 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594901+0000", 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9751e-05 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600445+0000", 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055442649999999996 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600512+0000", 2024-09-16T04:01:20.048 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6772000000000004e-05 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:132 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'90, mlcod=25'85)", 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.387929724999999, 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005744519, 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.049 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594815+0000", 2024-09-16T04:01:20.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2900000000000002e-07 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594825+0000", 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5640000000000002e-06 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594836+0000", 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1025e-05 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.051 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594844+0000", 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8699999999999998e-06 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595492+0000", 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00064781699999999997 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595549+0000", 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6842999999999998e-05 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.052 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600518+0000", 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004968386 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600559+0000", 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1185e-05 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:133 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'91, mlcod=25'85)", 2024-09-16T04:01:20.053 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.387875969, 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057530580000000001, 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:20.054 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.37e-07 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594874+0000", 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3890000000000004e-06 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594884+0000", 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8970000000000003e-06 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594890+0000", 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.5989999999999998e-06 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596024+0000", 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0011332289999999999 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.056 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596076+0000", 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2253000000000003e-05 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600563+0000", 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004487301 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600621+0000", 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7853000000000003e-05 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.057 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:134 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'92, mlcod=25'85)", 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.387393731, 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0068484870000000003, 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.058 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595352+0000", 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.059 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595414+0000", 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1926000000000003e-05 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595439+0000", 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4794999999999999e-05 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595453+0000", 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4283e-05 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.060 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596809+0000", 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001356407 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596847+0000", 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7731000000000002e-05 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602111+0000", 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052641040000000004 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.061 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602199+0000", 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7551000000000003e-05 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:135 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'93, mlcod=25'85)", 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.386724161, 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006226517, 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.062 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.063 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596021+0000", 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.403e-06 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596033+0000", 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1888999999999999e-05 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596051+0000", 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8091999999999999e-05 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.064 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596065+0000", 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4063000000000001e-05 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597461+0000", 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001395568 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597495+0000", 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3824e-05 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.065 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602202+0000", 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0047076369999999998 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602246+0000", 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4041000000000001e-05 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.066 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:136 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'94, mlcod=25'85)", 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.386222004, 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057697900000000003, 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:20.067 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596524+0000", 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.796e-06 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.068 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596536+0000", 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1963e-05 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596560+0000", 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3855000000000001e-05 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596572+0000", 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2836e-05 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.069 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597942+0000", 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0013696769999999999 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598001+0000", 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8856e-05 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602248+0000", 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042472500000000002 2024-09-16T04:01:20.070 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602292+0000", 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3557000000000003e-05 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.071 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:137 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'95, mlcod=25'85)", 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.596971+0000", 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.385772896999999, 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053633309999999998, 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596971+0000", 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.072 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596971+0000", 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596972+0000", 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1999999999999999e-06 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597033+0000", 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.0921999999999997e-05 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.073 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597051+0000", 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7448999999999998e-05 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597061+0000", 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.007e-05 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598891+0000", 2024-09-16T04:01:20.074 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0018303589999999999 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598927+0000", 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6168000000000003e-05 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602297+0000", 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003369698 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.075 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602334+0000", 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7465000000000001e-05 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:163 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'117, mlcod=25'112)", 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.203838415, 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061538799999999996, 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.076 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778907+0000", 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.08e-06 2024-09-16T04:01:20.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778932+0000", 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5661999999999999e-05 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778950+0000", 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7852000000000001e-05 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778961+0000", 2024-09-16T04:01:20.078 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1333e-05 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.779058+0000", 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6280000000000007e-05 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.779087+0000", 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9505e-05 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.784992+0000", 2024-09-16T04:01:20.079 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059048049999999999 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.785059+0000", 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7362999999999999e-05 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:164 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'118, mlcod=25'112)", 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.780311+0000", 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.202432822999999, 2024-09-16T04:01:20.080 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0047892610000000004, 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.780311+0000", 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.780311+0000", 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.780313+0000", 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6360000000000001e-06 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.780403+0000", 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.9853999999999997e-05 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.780438+0000", 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5315999999999999e-05 2024-09-16T04:01:20.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.780446+0000", 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7079999999999997e-06 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.780520+0000", 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4653999999999994e-05 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.780537+0000", 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7122e-05 2024-09-16T04:01:20.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.785062+0000", 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0045249859999999999 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.785100+0000", 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7985000000000003e-05 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.084 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:165 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'119, mlcod=25'112)", 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 20.201134926000002, 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056794660000000002, 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.085 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781611+0000", 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.827e-06 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781629+0000", 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7734999999999999e-05 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.086 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781657+0000", 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8065e-05 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781670+0000", 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2993000000000001e-05 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781828+0000", 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000158436 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.087 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781846+0000", 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7651000000000001e-05 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.787255+0000", 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054093639999999998 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.787288+0000", 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3395000000000001e-05 2024-09-16T04:01:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.089 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.089 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.089 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:20.089 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:20.089 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:20.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:01:20.091 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:20.086+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:20.191 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:20.186+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:20.219 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:20.219 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:20.219 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:20.219 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:20.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:01:20.293 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:20.291+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:20.358 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:20.358 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:20.358 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:20.358 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:20.358 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:20.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:01:20.394 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:20.393+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:20.495 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:20.494+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:20.538 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:20.538 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:20.538 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:20.538 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [] 2024-09-16T04:01:20.538 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:20.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:01:20.596 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:20.594+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:20.671 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:20.671 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:20.671 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:20.671 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:20.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:01:20.728 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:20.727+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:20.799 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:20.795+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:20.801 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:20.801 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:20.801 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:20.801 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:20.802 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:20.815 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:01:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T04:01:20.856 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T04:01:20.859 DEBUG:teuthology.orchestra.run.smithi188:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-16T04:01:20.881 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight read -> 100 2024-09-16T04:01:20.881 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight delete -> 50 2024-09-16T04:01:20.881 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight snap_create -> 50 2024-09-16T04:01:20.881 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight snap_remove -> 50 2024-09-16T04:01:20.881 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight rollback -> 50 2024-09-16T04:01:20.881 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight setattr -> 25 2024-09-16T04:01:20.881 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight rmattr -> 25 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight copy_from -> 50 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout:adding op weight append -> 100 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout:ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout:Configuration: 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout: Number of operations: 400 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout: Number of objects: 50 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout: Max in flight operations: 16 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout: Object size (in bytes): 4000000 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout: Write stride min: 400000 2024-09-16T04:01:20.882 INFO:tasks.rados.rados.0.smithi188.stdout: Write stride max: 800000 2024-09-16T04:01:20.899 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:20.894+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:20.983 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [] 2024-09-16T04:01:20.983 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:20.997 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:01:21.000 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:20.998+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:21.101 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:21.094+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:21.115 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:21.115 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:21.115 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:21.115 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:21.133 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:01:21.201 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:21.194+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:21.258 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:21.259 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:21.259 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:21.259 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:21.259 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:21.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:01:21.302 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:21.297+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:21.386 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:21.394 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:21.394 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:21.394 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:21.394 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [] 2024-09-16T04:01:21.394 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:21.408 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:01:21.433 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:21.429+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:21.504 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:21.498+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:21.539 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:21.539 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:21.541 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:21.541 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:21.541 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:01:21.541 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:21.556 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:01:21.560 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:21.582 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:21.582 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:21.602 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:21.605 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:21.602+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:21.623 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:21.623 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:21.646 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:21.646 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:21.670 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:21.670 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:21.680 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:21.680 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:21.693 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:21.693 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:21.694 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:01:21.705 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:21.697+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:21.715 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:21.715 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:21.739 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:21.739 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:21.761 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:21.762 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:21.781 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:21.803 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:21.803 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:21.805 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:21.802+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:21.817 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:21.817 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:21.826 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:21.826 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:21.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:21.821+0000 7fce89d9b640 0 -- 172.21.15.37:0/165053195 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7fce8c221820 msgr2=0x7fce8c223c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T04:01:21.832 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:01:21.835 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T04:01:21.841 INFO:tasks.rados.rados.0.smithi188.stdout:1: write initial oid 1 2024-09-16T04:01:21.841 INFO:tasks.rados.rados.0.smithi188.stdout:1: seq_num 0 ranges {0=32768} 2024-09-16T04:01:21.842 INFO:tasks.rados.rados.0.smithi188.stdout:1: writing smithi18831615-1 from 0 to 32768 tid 1 2024-09-16T04:01:21.842 INFO:tasks.rados.rados.0.smithi188.stdout:2: write initial oid 2 2024-09-16T04:01:21.842 INFO:tasks.rados.rados.0.smithi188.stdout:2: seq_num 1 ranges {0=32768} 2024-09-16T04:01:21.843 INFO:tasks.rados.rados.0.smithi188.stdout:2: writing smithi18831615-2 from 0 to 32768 tid 1 2024-09-16T04:01:21.843 INFO:tasks.rados.rados.0.smithi188.stdout:3: write initial oid 3 2024-09-16T04:01:21.843 INFO:tasks.rados.rados.0.smithi188.stdout:3: seq_num 2 ranges {0=49152} 2024-09-16T04:01:21.843 INFO:tasks.rados.rados.0.smithi188.stdout:3: writing smithi18831615-3 from 0 to 49152 tid 1 2024-09-16T04:01:21.844 INFO:tasks.rados.rados.0.smithi188.stdout:4: write initial oid 4 2024-09-16T04:01:21.844 INFO:tasks.rados.rados.0.smithi188.stdout:4: seq_num 3 ranges {0=65536} 2024-09-16T04:01:21.844 INFO:tasks.rados.rados.0.smithi188.stdout:4: writing smithi18831615-4 from 0 to 65536 tid 1 2024-09-16T04:01:21.844 INFO:tasks.rados.rados.0.smithi188.stdout:5: write initial oid 5 2024-09-16T04:01:21.844 INFO:tasks.rados.rados.0.smithi188.stdout:5: seq_num 4 ranges {0=81920} 2024-09-16T04:01:21.844 INFO:tasks.rados.rados.0.smithi188.stdout:5: writing smithi18831615-5 from 0 to 81920 tid 1 2024-09-16T04:01:21.844 INFO:tasks.rados.rados.0.smithi188.stdout:6: write initial oid 6 2024-09-16T04:01:21.844 INFO:tasks.rados.rados.0.smithi188.stdout:6: seq_num 5 ranges {0=98304} 2024-09-16T04:01:21.845 INFO:tasks.rados.rados.0.smithi188.stdout:6: writing smithi18831615-6 from 0 to 98304 tid 1 2024-09-16T04:01:21.845 INFO:tasks.rados.rados.0.smithi188.stdout:7: write initial oid 7 2024-09-16T04:01:21.845 INFO:tasks.rados.rados.0.smithi188.stdout:7: seq_num 6 ranges {0=114688} 2024-09-16T04:01:21.846 INFO:tasks.rados.rados.0.smithi188.stdout:7: writing smithi18831615-7 from 0 to 114688 tid 1 2024-09-16T04:01:21.846 INFO:tasks.rados.rados.0.smithi188.stdout:8: write initial oid 8 2024-09-16T04:01:21.847 INFO:tasks.rados.rados.0.smithi188.stdout:8: seq_num 7 ranges {0=131072} 2024-09-16T04:01:21.848 INFO:tasks.rados.rados.0.smithi188.stdout:8: writing smithi18831615-8 from 0 to 131072 tid 1 2024-09-16T04:01:21.848 INFO:tasks.rados.rados.0.smithi188.stdout:9: write initial oid 9 2024-09-16T04:01:21.848 INFO:tasks.rados.rados.0.smithi188.stdout:9: seq_num 8 ranges {0=147456} 2024-09-16T04:01:21.849 INFO:tasks.rados.rados.0.smithi188.stdout:9: writing smithi18831615-9 from 0 to 147456 tid 1 2024-09-16T04:01:21.850 INFO:tasks.rados.rados.0.smithi188.stdout:10: write initial oid 10 2024-09-16T04:01:21.851 INFO:tasks.rados.rados.0.smithi188.stdout:10: seq_num 9 ranges {0=163840} 2024-09-16T04:01:21.851 INFO:tasks.rados.rados.0.smithi188.stdout:10: writing smithi18831615-10 from 0 to 163840 tid 1 2024-09-16T04:01:21.851 INFO:tasks.rados.rados.0.smithi188.stdout:11: write initial oid 11 2024-09-16T04:01:21.851 INFO:tasks.rados.rados.0.smithi188.stdout:11: seq_num 10 ranges {0=180224} 2024-09-16T04:01:21.852 INFO:tasks.rados.rados.0.smithi188.stdout:11: writing smithi18831615-11 from 0 to 180224 tid 1 2024-09-16T04:01:21.853 INFO:tasks.rados.rados.0.smithi188.stdout:12: write initial oid 12 2024-09-16T04:01:21.853 INFO:tasks.rados.rados.0.smithi188.stdout:12: seq_num 11 ranges {0=196608} 2024-09-16T04:01:21.854 INFO:tasks.rados.rados.0.smithi188.stdout:12: writing smithi18831615-12 from 0 to 196608 tid 1 2024-09-16T04:01:21.854 INFO:tasks.rados.rados.0.smithi188.stdout:13: write initial oid 13 2024-09-16T04:01:21.854 INFO:tasks.rados.rados.0.smithi188.stdout:13: seq_num 12 ranges {0=212992} 2024-09-16T04:01:21.856 INFO:tasks.rados.rados.0.smithi188.stdout:13: writing smithi18831615-13 from 0 to 212992 tid 1 2024-09-16T04:01:21.856 INFO:tasks.rados.rados.0.smithi188.stdout:14: write initial oid 14 2024-09-16T04:01:21.856 INFO:tasks.rados.rados.0.smithi188.stdout:14: seq_num 13 ranges {0=229376} 2024-09-16T04:01:21.856 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:01:21.858 INFO:tasks.rados.rados.0.smithi188.stdout:14: writing smithi18831615-14 from 0 to 229376 tid 1 2024-09-16T04:01:21.858 INFO:tasks.rados.rados.0.smithi188.stdout:15: write initial oid 15 2024-09-16T04:01:21.858 INFO:tasks.rados.rados.0.smithi188.stdout:15: seq_num 14 ranges {0=245760} 2024-09-16T04:01:21.859 INFO:tasks.rados.rados.0.smithi188.stdout:15: writing smithi18831615-15 from 0 to 245760 tid 1 2024-09-16T04:01:21.860 INFO:tasks.rados.rados.0.smithi188.stdout:16: write initial oid 16 2024-09-16T04:01:21.860 INFO:tasks.rados.rados.0.smithi188.stdout:16: seq_num 15 ranges {0=262144} 2024-09-16T04:01:21.861 INFO:tasks.rados.rados.0.smithi188.stdout:16: writing smithi18831615-16 from 0 to 262144 tid 1 2024-09-16T04:01:21.861 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.862 INFO:tasks.rados.rados.0.smithi188.stdout:1: finishing write tid 1 to smithi18831615-1 2024-09-16T04:01:21.862 INFO:tasks.rados.rados.0.smithi188.stdout:7: finishing write tid 1 to smithi18831615-7 2024-09-16T04:01:21.863 INFO:tasks.rados.rados.0.smithi188.stdout:3: finishing write tid 1 to smithi18831615-3 2024-09-16T04:01:21.863 INFO:tasks.rados.rados.0.smithi188.stdout:2: finishing write tid 1 to smithi18831615-2 2024-09-16T04:01:21.863 INFO:tasks.rados.rados.0.smithi188.stdout:2: finishing write tid 2 to smithi18831615-2 2024-09-16T04:01:21.863 INFO:tasks.rados.rados.0.smithi188.stdout:1: finishing write tid 2 to smithi18831615-1 2024-09-16T04:01:21.867 INFO:tasks.rados.rados.0.smithi188.stdout:5: finishing write tid 1 to smithi18831615-5 2024-09-16T04:01:21.867 INFO:tasks.rados.rados.0.smithi188.stdout:8: finishing write tid 1 to smithi18831615-8 2024-09-16T04:01:21.867 INFO:tasks.rados.rados.0.smithi188.stdout:8: finishing write tid 2 to smithi18831615-8 2024-09-16T04:01:21.867 INFO:tasks.rados.rados.0.smithi188.stdout:13: finishing write tid 1 to smithi18831615-13 2024-09-16T04:01:21.867 INFO:tasks.rados.rados.0.smithi188.stdout:13: finishing write tid 2 to smithi18831615-13 2024-09-16T04:01:21.867 INFO:tasks.rados.rados.0.smithi188.stdout:7: finishing write tid 2 to smithi18831615-7 2024-09-16T04:01:21.867 INFO:tasks.rados.rados.0.smithi188.stdout:3: finishing write tid 2 to smithi18831615-3 2024-09-16T04:01:21.868 INFO:tasks.rados.rados.0.smithi188.stdout:5: finishing write tid 2 to smithi18831615-5 2024-09-16T04:01:21.870 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T04:01:21.871 INFO:tasks.rados.rados.0.smithi188.stdout:13: finishing write tid 3 to smithi18831615-13 2024-09-16T04:01:21.871 INFO:tasks.rados.rados.0.smithi188.stdout:13: oid 13 updating version 0 to 4 2024-09-16T04:01:21.871 INFO:tasks.rados.rados.0.smithi188.stdout:13: oid 13 version 4 is already newer than 3 2024-09-16T04:01:21.871 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T04:01:21.871 INFO:tasks.rados.rados.0.smithi188.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:01:21.871 INFO:tasks.rados.rados.0.smithi188.stdout:13: done (15 left) 2024-09-16T04:01:21.871 INFO:tasks.rados.rados.0.smithi188.stdout:17: write initial oid 17 2024-09-16T04:01:21.872 INFO:tasks.rados.rados.0.smithi188.stdout:17: seq_num 16 ranges {0=278528} 2024-09-16T04:01:21.873 INFO:tasks.rados.rados.0.smithi188.stdout:17: writing smithi18831615-17 from 0 to 278528 tid 1 2024-09-16T04:01:21.873 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.873 INFO:tasks.rados.rados.0.smithi188.stdout:8: finishing write tid 3 to smithi18831615-8 2024-09-16T04:01:21.873 INFO:tasks.rados.rados.0.smithi188.stdout:8: oid 8 updating version 0 to 1 2024-09-16T04:01:21.873 INFO:tasks.rados.rados.0.smithi188.stdout:8: oid 8 updating version 1 to 2 2024-09-16T04:01:21.873 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T04:01:21.873 INFO:tasks.rados.rados.0.smithi188.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:7: finishing write tid 3 to smithi18831615-7 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:7: oid 7 updating version 0 to 1 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:7: oid 7 updating version 1 to 2 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:9: finishing write tid 1 to smithi18831615-9 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:7: done (15 left) 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:8: done (14 left) 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:18: write initial oid 18 2024-09-16T04:01:21.874 INFO:tasks.rados.rados.0.smithi188.stdout:18: seq_num 17 ranges {0=294912} 2024-09-16T04:01:21.875 INFO:tasks.rados.rados.0.smithi188.stdout:18: writing smithi18831615-18 from 0 to 294912 tid 1 2024-09-16T04:01:21.875 INFO:tasks.rados.rados.0.smithi188.stdout:19: write initial oid 19 2024-09-16T04:01:21.875 INFO:tasks.rados.rados.0.smithi188.stdout:19: seq_num 18 ranges {0=311296} 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:19: writing smithi18831615-19 from 0 to 311296 tid 1 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:9: finishing write tid 2 to smithi18831615-9 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:6: finishing write tid 1 to smithi18831615-6 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:12: finishing write tid 1 to smithi18831615-12 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:6: finishing write tid 2 to smithi18831615-6 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:4: finishing write tid 1 to smithi18831615-4 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:5: finishing write tid 3 to smithi18831615-5 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:5: oid 5 updating version 0 to 1 2024-09-16T04:01:21.878 INFO:tasks.rados.rados.0.smithi188.stdout:5: oid 5 updating version 1 to 2 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:14: finishing write tid 1 to smithi18831615-14 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:3: finishing write tid 3 to smithi18831615-3 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:3: oid 3 updating version 0 to 1 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:3: oid 3 updating version 1 to 2 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:4: finishing write tid 2 to smithi18831615-4 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:2: finishing write tid 3 to smithi18831615-2 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:2: oid 2 updating version 0 to 1 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:2: oid 2 updating version 1 to 2 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T04:01:21.879 INFO:tasks.rados.rados.0.smithi188.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:9: finishing write tid 3 to smithi18831615-9 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:9: oid 9 updating version 0 to 4 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:9: oid 9 version 4 is already newer than 3 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 9 v 4 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:14: finishing write tid 2 to smithi18831615-14 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:6: finishing write tid 3 to smithi18831615-6 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:6: oid 6 updating version 0 to 1 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:6: oid 6 updating version 1 to 2 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:2: done (15 left) 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:3: done (14 left) 2024-09-16T04:01:21.880 INFO:tasks.rados.rados.0.smithi188.stdout:5: done (13 left) 2024-09-16T04:01:21.881 INFO:tasks.rados.rados.0.smithi188.stdout:6: done (12 left) 2024-09-16T04:01:21.881 INFO:tasks.rados.rados.0.smithi188.stdout:9: done (11 left) 2024-09-16T04:01:21.881 INFO:tasks.rados.rados.0.smithi188.stdout:20: write initial oid 20 2024-09-16T04:01:21.881 INFO:tasks.rados.rados.0.smithi188.stdout:20: seq_num 19 ranges {0=327680} 2024-09-16T04:01:21.881 INFO:tasks.rados.rados.0.smithi188.stdout:20: writing smithi18831615-20 from 0 to 327680 tid 1 2024-09-16T04:01:21.881 INFO:tasks.rados.rados.0.smithi188.stdout:21: write initial oid 21 2024-09-16T04:01:21.881 INFO:tasks.rados.rados.0.smithi188.stdout:21: seq_num 20 ranges {0=344064} 2024-09-16T04:01:21.882 INFO:tasks.rados.rados.0.smithi188.stdout:21: writing smithi18831615-21 from 0 to 344064 tid 1 2024-09-16T04:01:21.882 INFO:tasks.rados.rados.0.smithi188.stdout:22: write initial oid 22 2024-09-16T04:01:21.882 INFO:tasks.rados.rados.0.smithi188.stdout:22: seq_num 21 ranges {0=360448} 2024-09-16T04:01:21.885 INFO:tasks.rados.rados.0.smithi188.stdout:22: writing smithi18831615-22 from 0 to 360448 tid 1 2024-09-16T04:01:21.885 INFO:tasks.rados.rados.0.smithi188.stdout:23: write initial oid 23 2024-09-16T04:01:21.885 INFO:tasks.rados.rados.0.smithi188.stdout:23: seq_num 22 ranges {0=376832} 2024-09-16T04:01:21.888 INFO:tasks.rados.rados.0.smithi188.stdout:23: writing smithi18831615-23 from 0 to 376832 tid 1 2024-09-16T04:01:21.888 INFO:tasks.rados.rados.0.smithi188.stdout:24: write initial oid 24 2024-09-16T04:01:21.888 INFO:tasks.rados.rados.0.smithi188.stdout:24: seq_num 23 ranges {0=393216} 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout:24: writing smithi18831615-24 from 0 to 393216 tid 1 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout:12: finishing write tid 2 to smithi18831615-12 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout:15: finishing write tid 1 to smithi18831615-15 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout:15: finishing write tid 2 to smithi18831615-15 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout:16: finishing write tid 1 to smithi18831615-16 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout:12: finishing write tid 3 to smithi18831615-12 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout:12: oid 12 updating version 0 to 5 2024-09-16T04:01:21.891 INFO:tasks.rados.rados.0.smithi188.stdout:12: oid 12 updating version 5 to 6 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:15: finishing write tid 3 to smithi18831615-15 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:15: oid 15 updating version 0 to 3 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:15: oid 15 updating version 3 to 4 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:4: finishing write tid 3 to smithi18831615-4 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:4: oid 4 updating version 0 to 4 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:4: oid 4 version 4 is already newer than 3 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 4 v 4 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T04:01:21.892 INFO:tasks.rados.rados.0.smithi188.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:1: finishing write tid 3 to smithi18831615-1 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:1: oid 1 updating version 0 to 1 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:1: oid 1 updating version 1 to 2 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:11: finishing write tid 1 to smithi18831615-11 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:11: finishing write tid 2 to smithi18831615-11 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:1: done (15 left) 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:4: done (14 left) 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:12: done (13 left) 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:15: done (12 left) 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:25: write initial oid 25 2024-09-16T04:01:21.893 INFO:tasks.rados.rados.0.smithi188.stdout:25: seq_num 24 ranges {0=409600} 2024-09-16T04:01:21.894 INFO:tasks.rados.rados.0.smithi188.stdout:25: writing smithi18831615-25 from 0 to 409600 tid 1 2024-09-16T04:01:21.894 INFO:tasks.rados.rados.0.smithi188.stdout:26: write initial oid 26 2024-09-16T04:01:21.894 INFO:tasks.rados.rados.0.smithi188.stdout:26: seq_num 25 ranges {0=425984} 2024-09-16T04:01:21.896 INFO:tasks.rados.rados.0.smithi188.stdout:26: writing smithi18831615-26 from 0 to 425984 tid 1 2024-09-16T04:01:21.896 INFO:tasks.rados.rados.0.smithi188.stdout:27: write initial oid 27 2024-09-16T04:01:21.896 INFO:tasks.rados.rados.0.smithi188.stdout:27: seq_num 26 ranges {0=442368} 2024-09-16T04:01:21.899 INFO:tasks.rados.rados.0.smithi188.stdout:27: writing smithi18831615-27 from 0 to 442368 tid 1 2024-09-16T04:01:21.899 INFO:tasks.rados.rados.0.smithi188.stdout:28: write initial oid 28 2024-09-16T04:01:21.899 INFO:tasks.rados.rados.0.smithi188.stdout:28: seq_num 27 ranges {0=458752} 2024-09-16T04:01:21.903 INFO:tasks.rados.rados.0.smithi188.stdout:28: writing smithi18831615-28 from 0 to 458752 tid 1 2024-09-16T04:01:21.903 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.903 INFO:tasks.rados.rados.0.smithi188.stdout:14: finishing write tid 3 to smithi18831615-14 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:14: oid 14 updating version 0 to 1 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:14: oid 14 updating version 1 to 2 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:16: finishing write tid 2 to smithi18831615-16 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:18: finishing write tid 1 to smithi18831615-18 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:18: finishing write tid 2 to smithi18831615-18 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:11: finishing write tid 3 to smithi18831615-11 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:11: oid 11 updating version 0 to 5 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:11: oid 11 updating version 5 to 6 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-16T04:01:21.904 INFO:tasks.rados.rados.0.smithi188.stdout:21: finishing write tid 1 to smithi18831615-21 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:17: finishing write tid 1 to smithi18831615-17 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:16: finishing write tid 3 to smithi18831615-16 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:16: oid 16 updating version 0 to 8 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:16: oid 16 version 8 is already newer than 7 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 16 v 8 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:10: finishing write tid 1 to smithi18831615-10 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:10: finishing write tid 2 to smithi18831615-10 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:18: finishing write tid 3 to smithi18831615-18 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:18: oid 18 updating version 0 to 9 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:18: oid 18 updating version 9 to 10 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 18 v 10 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T04:01:21.905 INFO:tasks.rados.rados.0.smithi188.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:17: finishing write tid 2 to smithi18831615-17 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:20: finishing write tid 1 to smithi18831615-20 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:20: finishing write tid 2 to smithi18831615-20 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:21: finishing write tid 2 to smithi18831615-21 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:19: finishing write tid 1 to smithi18831615-19 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:21: finishing write tid 3 to smithi18831615-21 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:21: oid 21 updating version 0 to 8 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:21: oid 21 version 8 is already newer than 7 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 21 v 8 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:11: done (15 left) 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:14: done (14 left) 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:16: done (13 left) 2024-09-16T04:01:21.906 INFO:tasks.rados.rados.0.smithi188.stdout:18: done (12 left) 2024-09-16T04:01:21.907 INFO:tasks.rados.rados.0.smithi188.stdout:21: done (11 left) 2024-09-16T04:01:21.907 INFO:tasks.rados.rados.0.smithi188.stdout:29: write initial oid 29 2024-09-16T04:01:21.907 INFO:tasks.rados.rados.0.smithi188.stdout:29: seq_num 28 ranges {0=475136} 2024-09-16T04:01:21.908 INFO:tasks.rados.rados.0.smithi188.stdout:29: writing smithi18831615-29 from 0 to 475136 tid 1 2024-09-16T04:01:21.908 INFO:tasks.rados.rados.0.smithi188.stdout:30: write initial oid 30 2024-09-16T04:01:21.908 INFO:tasks.rados.rados.0.smithi188.stdout:30: seq_num 29 ranges {0=491520} 2024-09-16T04:01:21.909 INFO:tasks.rados.rados.0.smithi188.stdout:30: writing smithi18831615-30 from 0 to 491520 tid 1 2024-09-16T04:01:21.909 INFO:tasks.rados.rados.0.smithi188.stdout:31: write initial oid 31 2024-09-16T04:01:21.909 INFO:tasks.rados.rados.0.smithi188.stdout:31: seq_num 30 ranges {0=507904} 2024-09-16T04:01:21.909 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:21.906+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:21.912 INFO:tasks.rados.rados.0.smithi188.stdout:31: writing smithi18831615-31 from 0 to 507904 tid 1 2024-09-16T04:01:21.912 INFO:tasks.rados.rados.0.smithi188.stdout:32: write initial oid 32 2024-09-16T04:01:21.912 INFO:tasks.rados.rados.0.smithi188.stdout:32: seq_num 31 ranges {0=524288} 2024-09-16T04:01:21.916 INFO:tasks.rados.rados.0.smithi188.stdout:32: writing smithi18831615-32 from 0 to 524288 tid 1 2024-09-16T04:01:21.916 INFO:tasks.rados.rados.0.smithi188.stdout:33: write initial oid 33 2024-09-16T04:01:21.916 INFO:tasks.rados.rados.0.smithi188.stdout:33: seq_num 32 ranges {0=540672} 2024-09-16T04:01:21.920 INFO:tasks.rados.rados.0.smithi188.stdout:33: writing smithi18831615-33 from 0 to 540672 tid 1 2024-09-16T04:01:21.920 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.920 INFO:tasks.rados.rados.0.smithi188.stdout:22: finishing write tid 1 to smithi18831615-22 2024-09-16T04:01:21.920 INFO:tasks.rados.rados.0.smithi188.stdout:20: finishing write tid 3 to smithi18831615-20 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:20: oid 20 updating version 0 to 6 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:20: oid 20 version 6 is already newer than 5 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:22: finishing write tid 2 to smithi18831615-22 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:19: finishing write tid 2 to smithi18831615-19 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:10: finishing write tid 3 to smithi18831615-10 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:10: oid 10 updating version 0 to 3 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:10: oid 10 updating version 3 to 4 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 10 v 4 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:24: finishing write tid 1 to smithi18831615-24 2024-09-16T04:01:21.921 INFO:tasks.rados.rados.0.smithi188.stdout:24: finishing write tid 2 to smithi18831615-24 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:24: finishing write tid 3 to smithi18831615-24 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:24: oid 24 updating version 0 to 10 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:24: oid 24 version 10 is already newer than 9 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 24 v 10 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:17: finishing write tid 3 to smithi18831615-17 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:17: oid 17 updating version 0 to 2 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:17: oid 17 version 2 is already newer than 1 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:22: finishing write tid 3 to smithi18831615-22 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:22: oid 22 updating version 0 to 1 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:22: oid 22 updating version 1 to 2 2024-09-16T04:01:21.922 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 22 v 2 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:19: finishing write tid 3 to smithi18831615-19 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:19: oid 19 updating version 0 to 3 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:19: oid 19 updating version 3 to 4 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 19 v 4 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:26: finishing write tid 1 to smithi18831615-26 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:26: finishing write tid 2 to smithi18831615-26 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:23: finishing write tid 1 to smithi18831615-23 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:26: finishing write tid 3 to smithi18831615-26 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:26: oid 26 updating version 0 to 2 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:26: oid 26 version 2 is already newer than 1 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 26 v 2 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T04:01:21.923 INFO:tasks.rados.rados.0.smithi188.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:23: finishing write tid 2 to smithi18831615-23 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:29: finishing write tid 1 to smithi18831615-29 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:29: finishing write tid 2 to smithi18831615-29 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:28: finishing write tid 1 to smithi18831615-28 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:27: finishing write tid 1 to smithi18831615-27 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:28: finishing write tid 2 to smithi18831615-28 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:10: done (15 left) 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:17: done (14 left) 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:19: done (13 left) 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:20: done (12 left) 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:22: done (11 left) 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:24: done (10 left) 2024-09-16T04:01:21.924 INFO:tasks.rados.rados.0.smithi188.stdout:26: done (9 left) 2024-09-16T04:01:21.925 INFO:tasks.rados.rados.0.smithi188.stdout:34: write initial oid 34 2024-09-16T04:01:21.925 INFO:tasks.rados.rados.0.smithi188.stdout:34: seq_num 33 ranges {0=557056} 2024-09-16T04:01:21.925 INFO:tasks.rados.rados.0.smithi188.stdout:34: writing smithi18831615-34 from 0 to 557056 tid 1 2024-09-16T04:01:21.926 INFO:tasks.rados.rados.0.smithi188.stdout:35: write initial oid 35 2024-09-16T04:01:21.926 INFO:tasks.rados.rados.0.smithi188.stdout:35: seq_num 34 ranges {0=573440} 2024-09-16T04:01:21.927 INFO:tasks.rados.rados.0.smithi188.stdout:35: writing smithi18831615-35 from 0 to 573440 tid 1 2024-09-16T04:01:21.927 INFO:tasks.rados.rados.0.smithi188.stdout:36: write initial oid 36 2024-09-16T04:01:21.927 INFO:tasks.rados.rados.0.smithi188.stdout:36: seq_num 35 ranges {0=589824} 2024-09-16T04:01:21.931 INFO:tasks.rados.rados.0.smithi188.stdout:36: writing smithi18831615-36 from 0 to 589824 tid 1 2024-09-16T04:01:21.931 INFO:tasks.rados.rados.0.smithi188.stdout:37: write initial oid 37 2024-09-16T04:01:21.931 INFO:tasks.rados.rados.0.smithi188.stdout:37: seq_num 36 ranges {0=606208} 2024-09-16T04:01:21.935 INFO:tasks.rados.rados.0.smithi188.stdout:37: writing smithi18831615-37 from 0 to 606208 tid 1 2024-09-16T04:01:21.935 INFO:tasks.rados.rados.0.smithi188.stdout:38: write initial oid 38 2024-09-16T04:01:21.935 INFO:tasks.rados.rados.0.smithi188.stdout:38: seq_num 37 ranges {0=622592} 2024-09-16T04:01:21.939 INFO:tasks.rados.rados.0.smithi188.stdout:38: writing smithi18831615-38 from 0 to 622592 tid 1 2024-09-16T04:01:21.939 INFO:tasks.rados.rados.0.smithi188.stdout:39: write initial oid 39 2024-09-16T04:01:21.939 INFO:tasks.rados.rados.0.smithi188.stdout:39: seq_num 38 ranges {0=638976} 2024-09-16T04:01:21.943 INFO:tasks.rados.rados.0.smithi188.stdout:39: writing smithi18831615-39 from 0 to 638976 tid 1 2024-09-16T04:01:21.943 INFO:tasks.rados.rados.0.smithi188.stdout:40: write initial oid 40 2024-09-16T04:01:21.943 INFO:tasks.rados.rados.0.smithi188.stdout:40: seq_num 39 ranges {0=671744} 2024-09-16T04:01:21.948 INFO:tasks.rados.rados.0.smithi188.stdout:40: writing smithi18831615-40 from 0 to 671744 tid 1 2024-09-16T04:01:21.948 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.948 INFO:tasks.rados.rados.0.smithi188.stdout:27: finishing write tid 2 to smithi18831615-27 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:29: finishing write tid 3 to smithi18831615-29 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:29: oid 29 updating version 0 to 5 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:29: oid 29 updating version 5 to 6 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:23: finishing write tid 3 to smithi18831615-23 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:23: oid 23 updating version 0 to 2 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:23: oid 23 version 2 is already newer than 1 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 23 v 2 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T04:01:21.949 INFO:tasks.rados.rados.0.smithi188.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:31: finishing write tid 1 to smithi18831615-31 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:30: finishing write tid 1 to smithi18831615-30 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:31: finishing write tid 2 to smithi18831615-31 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:25: finishing write tid 1 to smithi18831615-25 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:25: finishing write tid 2 to smithi18831615-25 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:30: finishing write tid 2 to smithi18831615-30 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:32: finishing write tid 1 to smithi18831615-32 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:32: finishing write tid 2 to smithi18831615-32 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:31: finishing write tid 3 to smithi18831615-31 2024-09-16T04:01:21.950 INFO:tasks.rados.rados.0.smithi188.stdout:31: oid 31 updating version 0 to 7 2024-09-16T04:01:21.952 INFO:tasks.rados.rados.0.smithi188.stdout:31: oid 31 updating version 7 to 8 2024-09-16T04:01:21.952 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 31 v 8 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:30: finishing write tid 3 to smithi18831615-30 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:30: oid 30 updating version 0 to 2 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:30: oid 30 version 2 is already newer than 1 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 30 v 2 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:23: done (15 left) 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:29: done (14 left) 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:30: done (13 left) 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:31: done (12 left) 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:41: write initial oid 41 2024-09-16T04:01:21.953 INFO:tasks.rados.rados.0.smithi188.stdout:41: seq_num 40 ranges {0=688128} 2024-09-16T04:01:21.954 INFO:tasks.rados.rados.0.smithi188.stdout:41: writing smithi18831615-41 from 0 to 688128 tid 1 2024-09-16T04:01:21.954 INFO:tasks.rados.rados.0.smithi188.stdout:42: write initial oid 42 2024-09-16T04:01:21.954 INFO:tasks.rados.rados.0.smithi188.stdout:42: seq_num 41 ranges {0=704512} 2024-09-16T04:01:21.956 INFO:tasks.rados.rados.0.smithi188.stdout:42: writing smithi18831615-42 from 0 to 704512 tid 1 2024-09-16T04:01:21.957 INFO:tasks.rados.rados.0.smithi188.stdout:43: write initial oid 43 2024-09-16T04:01:21.957 INFO:tasks.rados.rados.0.smithi188.stdout:43: seq_num 42 ranges {0=720896} 2024-09-16T04:01:21.960 INFO:tasks.rados.rados.0.smithi188.stdout:43: writing smithi18831615-43 from 0 to 720896 tid 1 2024-09-16T04:01:21.960 INFO:tasks.rados.rados.0.smithi188.stdout:44: write initial oid 44 2024-09-16T04:01:21.961 INFO:tasks.rados.rados.0.smithi188.stdout:44: seq_num 43 ranges {0=737280} 2024-09-16T04:01:21.965 INFO:tasks.rados.rados.0.smithi188.stdout:44: writing smithi18831615-44 from 0 to 737280 tid 1 2024-09-16T04:01:21.965 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.965 INFO:tasks.rados.rados.0.smithi188.stdout:28: finishing write tid 3 to smithi18831615-28 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:28: oid 28 updating version 0 to 4 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:28: oid 28 version 4 is already newer than 3 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 28 v 4 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:25: finishing write tid 3 to smithi18831615-25 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:25: oid 25 updating version 0 to 4 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:25: oid 25 version 4 is already newer than 3 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-16T04:01:21.966 INFO:tasks.rados.rados.0.smithi188.stdout:32: finishing write tid 3 to smithi18831615-32 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:32: oid 32 updating version 0 to 3 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:32: oid 32 updating version 3 to 4 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 32 v 4 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:27: finishing write tid 3 to smithi18831615-27 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:27: oid 27 updating version 0 to 2 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:27: oid 27 version 2 is already newer than 1 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 27 v 2 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:33: finishing write tid 1 to smithi18831615-33 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:25: done (15 left) 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:27: done (14 left) 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:28: done (13 left) 2024-09-16T04:01:21.967 INFO:tasks.rados.rados.0.smithi188.stdout:32: done (12 left) 2024-09-16T04:01:21.968 INFO:tasks.rados.rados.0.smithi188.stdout:45: write initial oid 45 2024-09-16T04:01:21.968 INFO:tasks.rados.rados.0.smithi188.stdout:45: seq_num 44 ranges {0=753664} 2024-09-16T04:01:21.970 INFO:tasks.rados.rados.0.smithi188.stdout:45: writing smithi18831615-45 from 0 to 753664 tid 1 2024-09-16T04:01:21.970 INFO:tasks.rados.rados.0.smithi188.stdout:46: write initial oid 46 2024-09-16T04:01:21.970 INFO:tasks.rados.rados.0.smithi188.stdout:46: seq_num 45 ranges {0=770048} 2024-09-16T04:01:21.975 INFO:tasks.rados.rados.0.smithi188.stdout:46: writing smithi18831615-46 from 0 to 770048 tid 1 2024-09-16T04:01:21.975 INFO:tasks.rados.rados.0.smithi188.stdout:47: write initial oid 47 2024-09-16T04:01:21.975 INFO:tasks.rados.rados.0.smithi188.stdout:47: seq_num 46 ranges {0=786432} 2024-09-16T04:01:21.980 INFO:tasks.rados.rados.0.smithi188.stdout:47: writing smithi18831615-47 from 0 to 786432 tid 1 2024-09-16T04:01:21.980 INFO:tasks.rados.rados.0.smithi188.stdout:48: write initial oid 48 2024-09-16T04:01:21.980 INFO:tasks.rados.rados.0.smithi188.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-16T04:01:21.983 INFO:tasks.rados.rados.0.smithi188.stdout:48: writing smithi18831615-48 from 0 to 425984 tid 1 2024-09-16T04:01:21.985 INFO:tasks.rados.rados.0.smithi188.stdout:48: writing smithi18831615-48 from 425984 to 802816 tid 2 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:33: finishing write tid 2 to smithi18831615-33 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:34: finishing write tid 1 to smithi18831615-34 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:34: finishing write tid 2 to smithi18831615-34 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:33: finishing write tid 3 to smithi18831615-33 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:33: oid 33 updating version 0 to 7 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:33: oid 33 updating version 7 to 8 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 33 v 8 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:34: finishing write tid 3 to smithi18831615-34 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:34: oid 34 updating version 0 to 3 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:34: oid 34 updating version 3 to 4 2024-09-16T04:01:21.986 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 34 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T04:01:21.987 INFO:tasks.rados.rados.0.smithi188.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-16T04:01:21.987 INFO:tasks.rados.rados.0.smithi188.stdout:35: finishing write tid 1 to smithi18831615-35 2024-09-16T04:01:21.987 INFO:tasks.rados.rados.0.smithi188.stdout:35: finishing write tid 2 to smithi18831615-35 2024-09-16T04:01:21.987 INFO:tasks.rados.rados.0.smithi188.stdout:33: done (15 left) 2024-09-16T04:01:21.987 INFO:tasks.rados.rados.0.smithi188.stdout:34: done (14 left) 2024-09-16T04:01:21.987 INFO:tasks.rados.rados.0.smithi188.stdout:49: write initial oid 49 2024-09-16T04:01:21.987 INFO:tasks.rados.rados.0.smithi188.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-16T04:01:21.988 INFO:tasks.rados.rados.0.smithi188.stdout:49: writing smithi18831615-49 from 0 to 442368 tid 1 2024-09-16T04:01:21.990 INFO:tasks.rados.rados.0.smithi188.stdout:49: writing smithi18831615-49 from 442368 to 819200 tid 2 2024-09-16T04:01:21.990 INFO:tasks.rados.rados.0.smithi188.stdout:50: write initial oid 50 2024-09-16T04:01:21.991 INFO:tasks.rados.rados.0.smithi188.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-16T04:01:21.993 INFO:tasks.rados.rados.0.smithi188.stdout:50: writing smithi18831615-50 from 0 to 458752 tid 1 2024-09-16T04:01:21.998 INFO:tasks.rados.rados.0.smithi188.stdout:50: writing smithi18831615-50 from 458752 to 835584 tid 2 2024-09-16T04:01:21.998 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 16 2024-09-16T04:01:21.998 INFO:tasks.rados.rados.0.smithi188.stdout:36: finishing write tid 1 to smithi18831615-36 2024-09-16T04:01:21.998 INFO:tasks.rados.rados.0.smithi188.stdout:35: finishing write tid 3 to smithi18831615-35 2024-09-16T04:01:21.998 INFO:tasks.rados.rados.0.smithi188.stdout:35: oid 35 updating version 0 to 5 2024-09-16T04:01:21.998 INFO:tasks.rados.rados.0.smithi188.stdout:35: oid 35 updating version 5 to 6 2024-09-16T04:01:21.998 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:36: finishing write tid 2 to smithi18831615-36 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:37: finishing write tid 1 to smithi18831615-37 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:37: finishing write tid 2 to smithi18831615-37 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:37: finishing write tid 3 to smithi18831615-37 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:37: oid 37 updating version 0 to 11 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:37: oid 37 updating version 11 to 12 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 37 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:36: finishing write tid 3 to smithi18831615-36 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:36: oid 36 updating version 0 to 1 2024-09-16T04:01:21.999 INFO:tasks.rados.rados.0.smithi188.stdout:36: oid 36 updating version 1 to 2 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 36 v 2 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:38: finishing write tid 1 to smithi18831615-38 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:38: finishing write tid 2 to smithi18831615-38 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:35: done (15 left) 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:36: done (14 left) 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:37: done (13 left) 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:51: rollback oid 37 current snap is 0 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:51: done (13 left) 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:52: rollback oid 34 current snap is 0 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:52: done (13 left) 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:53: delete oid 33 current snap is 0 2024-09-16T04:01:22.000 INFO:tasks.rados.rados.0.smithi188.stdout:38: finishing write tid 3 to smithi18831615-38 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:38: oid 38 updating version 0 to 13 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:38: oid 38 updating version 13 to 14 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 38 v 14 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:39: finishing write tid 1 to smithi18831615-39 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:39: finishing write tid 2 to smithi18831615-39 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:39: finishing write tid 3 to smithi18831615-39 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:39: oid 39 updating version 0 to 12 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:39: oid 39 version 12 is already newer than 11 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:40: finishing write tid 1 to smithi18831615-40 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:40: finishing write tid 2 to smithi18831615-40 2024-09-16T04:01:22.001 INFO:tasks.rados.rados.0.smithi188.stdout:40: finishing write tid 3 to smithi18831615-40 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:40: oid 40 updating version 0 to 5 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:40: oid 40 updating version 5 to 6 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:41: finishing write tid 1 to smithi18831615-41 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:41: finishing write tid 2 to smithi18831615-41 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:41: finishing write tid 3 to smithi18831615-41 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:41: oid 41 updating version 0 to 3 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:41: oid 41 updating version 3 to 4 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 41 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:42: finishing write tid 1 to smithi18831615-42 2024-09-16T04:01:22.002 INFO:tasks.rados.rados.0.smithi188.stdout:43: finishing write tid 1 to smithi18831615-43 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:42: finishing write tid 2 to smithi18831615-42 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:43: finishing write tid 2 to smithi18831615-43 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:43: finishing write tid 3 to smithi18831615-43 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:43: oid 43 updating version 0 to 9 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:43: oid 43 updating version 9 to 10 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 43 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:42: finishing write tid 3 to smithi18831615-42 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:42: oid 42 updating version 0 to 1 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:42: oid 42 updating version 1 to 2 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 42 v 2 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:44: finishing write tid 1 to smithi18831615-44 2024-09-16T04:01:22.003 INFO:tasks.rados.rados.0.smithi188.stdout:44: finishing write tid 2 to smithi18831615-44 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:44: finishing write tid 3 to smithi18831615-44 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:44: oid 44 updating version 0 to 3 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:44: oid 44 updating version 3 to 4 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 44 v 4 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:45: finishing write tid 1 to smithi18831615-45 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:45: finishing write tid 2 to smithi18831615-45 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:46: finishing write tid 1 to smithi18831615-46 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:46: finishing write tid 2 to smithi18831615-46 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:45: finishing write tid 3 to smithi18831615-45 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:45: oid 45 updating version 0 to 5 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:45: oid 45 updating version 5 to 6 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T04:01:22.004 INFO:tasks.rados.rados.0.smithi188.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:46: finishing write tid 3 to smithi18831615-46 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:46: oid 46 updating version 0 to 7 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:46: oid 46 updating version 7 to 8 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 46 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:47: finishing write tid 1 to smithi18831615-47 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:47: finishing write tid 2 to smithi18831615-47 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:48: finishing write tid 1 to smithi18831615-48 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:47: finishing write tid 3 to smithi18831615-47 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:47: oid 47 updating version 0 to 4 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:47: oid 47 version 4 is already newer than 3 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 47 v 4 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T04:01:22.005 INFO:tasks.rados.rados.0.smithi188.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-16T04:01:22.006 INFO:tasks.rados.rados.0.smithi188.stdout:48: finishing write tid 2 to smithi18831615-48 2024-09-16T04:01:22.006 INFO:tasks.rados.rados.0.smithi188.stdout:48: finishing write tid 3 to smithi18831615-48 2024-09-16T04:01:22.006 INFO:tasks.rados.rados.0.smithi188.stdout:49: finishing write tid 1 to smithi18831615-49 2024-09-16T04:01:22.006 INFO:tasks.rados.rados.0.smithi188.stdout:49: finishing write tid 2 to smithi18831615-49 2024-09-16T04:01:22.006 INFO:tasks.rados.rados.0.smithi188.stdout:48: finishing write tid 4 to smithi18831615-48 2024-09-16T04:01:22.006 INFO:tasks.rados.rados.0.smithi188.stdout:48: oid 48 updating version 0 to 16 2024-09-16T04:01:22.007 INFO:tasks.rados.rados.0.smithi188.stdout:48: oid 48 version 16 is already newer than 15 2024-09-16T04:01:22.007 INFO:tasks.rados.rados.0.smithi188.stdout:48: oid 48 updating version 16 to 17 2024-09-16T04:01:22.007 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 48 v 17 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T04:01:22.007 INFO:tasks.rados.rados.0.smithi188.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-16T04:01:22.007 INFO:tasks.rados.rados.0.smithi188.stdout:49: finishing write tid 3 to smithi18831615-49 2024-09-16T04:01:22.007 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:22.007 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:50: finishing write tid 1 to smithi18831615-50 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:50: finishing write tid 2 to smithi18831615-50 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:50: finishing write tid 3 to smithi18831615-50 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:49: finishing write tid 4 to smithi18831615-49 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:49: oid 49 updating version 0 to 10 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:49: oid 49 version 10 is already newer than 9 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:49: oid 49 updating version 10 to 11 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 49 v 11 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:50: finishing write tid 4 to smithi18831615-50 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:50: oid 50 updating version 0 to 10 2024-09-16T04:01:22.009 INFO:tasks.rados.rados.0.smithi188.stdout:50: oid 50 updating version 10 to 11 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:50: oid 50 version 11 is already newer than 9 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 50 v 11 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:38: done (13 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:39: done (12 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:40: done (11 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:41: done (10 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:42: done (9 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:43: done (8 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:44: done (7 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:45: done (6 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:46: done (5 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:47: done (4 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:48: done (3 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:49: done (2 left) 2024-09-16T04:01:22.010 INFO:tasks.rados.rados.0.smithi188.stdout:50: done (1 left) 2024-09-16T04:01:22.011 INFO:tasks.rados.rados.0.smithi188.stdout:53: done (0 left) 2024-09-16T04:01:22.011 INFO:tasks.rados.rados.0.smithi188.stdout:54: read oid 5 snap -1 2024-09-16T04:01:22.011 INFO:tasks.rados.rados.0.smithi188.stdout:54: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:01:22.011 INFO:tasks.rados.rados.0.smithi188.stdout:55: delete oid 11 current snap is 0 2024-09-16T04:01:22.011 INFO:tasks.rados.rados.0.smithi188.stdout:54: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:01:22.012 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:22.005+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:22.013 INFO:tasks.rados.rados.0.smithi188.stdout:54: done (1 left) 2024-09-16T04:01:22.013 INFO:tasks.rados.rados.0.smithi188.stdout:55: done (0 left) 2024-09-16T04:01:22.013 INFO:tasks.rados.rados.0.smithi188.stdout:56: read oid 43 snap -1 2024-09-16T04:01:22.013 INFO:tasks.rados.rados.0.smithi188.stdout:56: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:01:22.013 INFO:tasks.rados.rados.0.smithi188.stdout:57: rmattr oid 1 current snap is 0 2024-09-16T04:01:22.013 INFO:tasks.rados.rados.0.smithi188.stdout:57: done (1 left) 2024-09-16T04:01:22.013 INFO:tasks.rados.rados.0.smithi188.stdout:58: delete oid 31 current snap is 0 2024-09-16T04:01:22.017 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:01:22.018 INFO:tasks.rados.rados.0.smithi188.stdout:58: done (1 left) 2024-09-16T04:01:22.019 INFO:tasks.rados.rados.0.smithi188.stdout:59: read oid 26 snap -1 2024-09-16T04:01:22.019 INFO:tasks.rados.rados.0.smithi188.stdout:59: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-16T04:01:22.019 INFO:tasks.rados.rados.0.smithi188.stdout:60: read oid 18 snap -1 2024-09-16T04:01:22.019 INFO:tasks.rados.rados.0.smithi188.stdout:60: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T04:01:22.019 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 36 current snap is 0 2024-09-16T04:01:22.019 INFO:tasks.rados.rados.0.smithi188.stdout:61: seq_num 53 ranges {589824=524288,1114112=376832} 2024-09-16T04:01:22.025 INFO:tasks.rados.rados.0.smithi188.stdout:61: writing smithi18831615-36 from 589824 to 1114112 tid 1 2024-09-16T04:01:22.027 INFO:tasks.rados.rados.0.smithi188.stdout:61: writing smithi18831615-36 from 1114112 to 1490944 tid 2 2024-09-16T04:01:22.028 INFO:tasks.rados.rados.0.smithi188.stdout:62: setattr oid 45 current snap is 0 2024-09-16T04:01:22.028 INFO:tasks.rados.rados.0.smithi188.stdout:56: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:01:22.048 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:22.048 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:22.050 INFO:tasks.rados.rados.0.smithi188.stdout:60: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T04:01:22.059 INFO:tasks.rados.rados.0.smithi188.stdout:59: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-16T04:01:22.072 INFO:tasks.rados.rados.0.smithi188.stdout:61: finishing write tid 1 to smithi18831615-36 2024-09-16T04:01:22.072 INFO:tasks.rados.rados.0.smithi188.stdout:61: finishing write tid 2 to smithi18831615-36 2024-09-16T04:01:22.072 INFO:tasks.rados.rados.0.smithi188.stdout:61: finishing write tid 3 to smithi18831615-36 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:61: finishing write tid 4 to smithi18831615-36 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:61: oid 36 updating version 0 to 7 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:61: oid 36 version 7 is already newer than 6 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:61: oid 36 version 7 is already newer than 5 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 36 v 7 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:61: left oid 36 (ObjNum 53 snap 0 seq_num 53) 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:56: done (4 left) 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:59: done (3 left) 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:60: done (2 left) 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:61: done (1 left) 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:63: copy_from oid 34 from oid 37 current snap is 0 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:64: read oid 43 snap -1 2024-09-16T04:01:22.073 INFO:tasks.rados.rados.0.smithi188.stdout:64: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:01:22.074 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 16 current snap is 0 2024-09-16T04:01:22.074 INFO:tasks.rados.rados.0.smithi188.stdout:65: seq_num 54 ranges {262144=540672,802816=376832} 2024-09-16T04:01:22.078 INFO:tasks.rados.rados.0.smithi188.stdout:65: writing smithi18831615-16 from 262144 to 802816 tid 1 2024-09-16T04:01:22.081 INFO:tasks.rados.rados.0.smithi188.stdout:65: writing smithi18831615-16 from 802816 to 1179648 tid 2 2024-09-16T04:01:22.082 INFO:tasks.rados.rados.0.smithi188.stdout:66: read oid 21 snap -1 2024-09-16T04:01:22.082 INFO:tasks.rados.rados.0.smithi188.stdout:66: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T04:01:22.082 INFO:tasks.rados.rados.0.smithi188.stdout:67: copy_from oid 3 from oid 41 current snap is 0 2024-09-16T04:01:22.082 INFO:tasks.rados.rados.0.smithi188.stdout:63: finishing copy_from racing read to smithi18831615-34 2024-09-16T04:01:22.082 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 6 current snap is 0 2024-09-16T04:01:22.082 INFO:tasks.rados.rados.0.smithi188.stdout:68: seq_num 55 ranges {98304=557056,655360=376832} 2024-09-16T04:01:22.084 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:22.084 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:22.086 INFO:tasks.rados.rados.0.smithi188.stdout:68: writing smithi18831615-6 from 98304 to 655360 tid 1 2024-09-16T04:01:22.090 INFO:tasks.rados.rados.0.smithi188.stdout:68: writing smithi18831615-6 from 655360 to 1032192 tid 2 2024-09-16T04:01:22.090 INFO:tasks.rados.rados.0.smithi188.stdout:69: read oid 26 snap -1 2024-09-16T04:01:22.090 INFO:tasks.rados.rados.0.smithi188.stdout:69: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-16T04:01:22.090 INFO:tasks.rados.rados.0.smithi188.stdout:70: setattr oid 18 current snap is 0 2024-09-16T04:01:22.090 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 45 v 7 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T04:01:22.090 INFO:tasks.rados.rados.0.smithi188.stdout:64: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:01:22.108 INFO:tasks.rados.rados.0.smithi188.stdout:63: finishing copy_from to smithi18831615-34 2024-09-16T04:01:22.109 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 34 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:01:22.109 INFO:tasks.rados.rados.0.smithi188.stdout:66: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T04:01:22.113 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:22.106+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:22.118 INFO:tasks.rados.rados.0.smithi188.stdout:67: finishing copy_from racing read to smithi18831615-3 2024-09-16T04:01:22.118 INFO:tasks.rados.rados.0.smithi188.stdout:67: finishing copy_from to smithi18831615-3 2024-09-16T04:01:22.118 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 3 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T04:01:22.118 INFO:tasks.rados.rados.0.smithi188.stdout:65: finishing write tid 1 to smithi18831615-16 2024-09-16T04:01:22.118 INFO:tasks.rados.rados.0.smithi188.stdout:68: finishing write tid 1 to smithi18831615-6 2024-09-16T04:01:22.118 INFO:tasks.rados.rados.0.smithi188.stdout:69: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-16T04:01:22.129 INFO:tasks.rados.rados.0.smithi188.stdout:65: finishing write tid 2 to smithi18831615-16 2024-09-16T04:01:22.129 INFO:tasks.rados.rados.0.smithi188.stdout:68: finishing write tid 2 to smithi18831615-6 2024-09-16T04:01:22.129 INFO:tasks.rados.rados.0.smithi188.stdout:65: finishing write tid 3 to smithi18831615-16 2024-09-16T04:01:22.129 INFO:tasks.rados.rados.0.smithi188.stdout:68: finishing write tid 3 to smithi18831615-6 2024-09-16T04:01:22.129 INFO:tasks.rados.rados.0.smithi188.stdout:68: finishing write tid 4 to smithi18831615-6 2024-09-16T04:01:22.129 INFO:tasks.rados.rados.0.smithi188.stdout:68: oid 6 updating version 0 to 5 2024-09-16T04:01:22.129 INFO:tasks.rados.rados.0.smithi188.stdout:68: oid 6 updating version 5 to 6 2024-09-16T04:01:22.129 INFO:tasks.rados.rados.0.smithi188.stdout:68: oid 6 updating version 6 to 7 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 6 v 7 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:68: left oid 6 (ObjNum 55 snap 0 seq_num 55) 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:65: finishing write tid 4 to smithi18831615-16 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:65: oid 16 updating version 0 to 14 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:65: oid 16 version 14 is already newer than 13 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:65: oid 16 updating version 14 to 15 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 16 v 15 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:65: left oid 16 (ObjNum 54 snap 0 seq_num 54) 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 18 v 16 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:62: done (8 left) 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:63: done (7 left) 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:64: done (6 left) 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:65: done (5 left) 2024-09-16T04:01:22.130 INFO:tasks.rados.rados.0.smithi188.stdout:66: done (4 left) 2024-09-16T04:01:22.131 INFO:tasks.rados.rados.0.smithi188.stdout:67: done (3 left) 2024-09-16T04:01:22.131 INFO:tasks.rados.rados.0.smithi188.stdout:68: done (2 left) 2024-09-16T04:01:22.131 INFO:tasks.rados.rados.0.smithi188.stdout:69: done (1 left) 2024-09-16T04:01:22.131 INFO:tasks.rados.rados.0.smithi188.stdout:70: done (0 left) 2024-09-16T04:01:22.131 INFO:tasks.rados.rados.0.smithi188.stdout:71: rollback oid 42 current snap is 0 2024-09-16T04:01:22.131 INFO:tasks.rados.rados.0.smithi188.stdout:71: done (0 left) 2024-09-16T04:01:22.131 INFO:tasks.rados.rados.0.smithi188.stdout:72: delete oid 40 current snap is 0 2024-09-16T04:01:22.132 INFO:tasks.rados.rados.0.smithi188.stdout:72: done (0 left) 2024-09-16T04:01:22.132 INFO:tasks.rados.rados.0.smithi188.stdout:73: delete oid 25 current snap is 0 2024-09-16T04:01:22.137 INFO:tasks.rados.rados.0.smithi188.stdout:73: done (0 left) 2024-09-16T04:01:22.137 INFO:tasks.rados.rados.0.smithi188.stdout:74: read oid 34 snap -1 2024-09-16T04:01:22.137 INFO:tasks.rados.rados.0.smithi188.stdout:74: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:01:22.137 INFO:tasks.rados.rados.0.smithi188.stdout:75: rollback oid 35 current snap is 0 2024-09-16T04:01:22.137 INFO:tasks.rados.rados.0.smithi188.stdout:75: done (1 left) 2024-09-16T04:01:22.137 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 4 current snap is 0 2024-09-16T04:01:22.137 INFO:tasks.rados.rados.0.smithi188.stdout:76: seq_num 58 ranges {65536=606208,671744=376832} 2024-09-16T04:01:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:76: writing smithi18831615-4 from 65536 to 671744 tid 1 2024-09-16T04:01:22.145 INFO:tasks.rados.rados.0.smithi188.stdout:76: writing smithi18831615-4 from 671744 to 1048576 tid 2 2024-09-16T04:01:22.145 INFO:tasks.rados.rados.0.smithi188.stdout:77: read oid 23 snap -1 2024-09-16T04:01:22.145 INFO:tasks.rados.rados.0.smithi188.stdout:77: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:01:22.145 INFO:tasks.rados.rados.0.smithi188.stdout:78: snap_create 2024-09-16T04:01:22.145 INFO:tasks.rados.rados.0.smithi188.stdout:74: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:01:22.157 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:22.157 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:22.161 INFO:tasks.rados.rados.0.smithi188.stdout:76: finishing write tid 1 to smithi18831615-4 2024-09-16T04:01:22.161 INFO:tasks.rados.rados.0.smithi188.stdout:76: finishing write tid 2 to smithi18831615-4 2024-09-16T04:01:22.161 INFO:tasks.rados.rados.0.smithi188.stdout:76: finishing write tid 3 to smithi18831615-4 2024-09-16T04:01:22.162 INFO:tasks.rados.rados.0.smithi188.stdout:76: finishing write tid 4 to smithi18831615-4 2024-09-16T04:01:22.162 INFO:tasks.rados.rados.0.smithi188.stdout:76: oid 4 updating version 0 to 21 2024-09-16T04:01:22.162 INFO:tasks.rados.rados.0.smithi188.stdout:76: oid 4 version 21 is already newer than 20 2024-09-16T04:01:22.162 INFO:tasks.rados.rados.0.smithi188.stdout:76: oid 4 version 21 is already newer than 19 2024-09-16T04:01:22.162 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 4 v 21 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-16T04:01:22.162 INFO:tasks.rados.rados.0.smithi188.stdout:76: left oid 4 (ObjNum 58 snap 0 seq_num 58) 2024-09-16T04:01:22.162 INFO:tasks.rados.rados.0.smithi188.stdout:77: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:01:22.171 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:01:22.214 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:22.210+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:22.315 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:22.314+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:22.332 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:22.332 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:22.346 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:01:22.416 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:22.414+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:22.468 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:22.468 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:22.481 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:01:22.517 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:22.513+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:22.606 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:22.606 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:22.617 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:22.614+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:22.619 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:01:22.718 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:22.714+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:22.742 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:22.742 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:22.757 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:01:22.820 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:22.813+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:22.839 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T04:01:22.844 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:74: done (3 left) 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:76: done (2 left) 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:77: done (1 left) 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:78: done (0 left) 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:79: read oid 15 snap 0 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:79: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:80: copy_from oid 9 from oid 33 current snap is 1 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 37 current snap is 1 2024-09-16T04:01:22.845 INFO:tasks.rados.rados.0.smithi188.stdout:81: seq_num 59 ranges {606208=622592,1228800=376832} 2024-09-16T04:01:22.860 INFO:tasks.rados.rados.0.smithi188.stdout:81: writing smithi18831615-37 from 606208 to 1228800 tid 1 2024-09-16T04:01:22.865 INFO:tasks.rados.rados.0.smithi188.stdout:81: writing smithi18831615-37 from 1228800 to 1605632 tid 2 2024-09-16T04:01:22.866 INFO:tasks.rados.rados.0.smithi188.stdout:82: setattr oid 30 current snap is 1 2024-09-16T04:01:22.866 INFO:tasks.rados.rados.0.smithi188.stdout:79: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T04:01:22.870 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T04:01:22.875 INFO:tasks.rados.rados.0.smithi188.stdout:80: finishing copy_from to smithi18831615-9 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:80: got expected ENOENT (src dne) 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:80: finishing copy_from racing read to smithi18831615-9 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: finishing write tid 1 to smithi18831615-37 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: finishing write tid 2 to smithi18831615-37 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: finishing write tid 3 to smithi18831615-37 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: finishing write tid 4 to smithi18831615-37 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: oid 37 updating version 0 to 25 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: oid 37 version 25 is already newer than 24 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: oid 37 version 25 is already newer than 23 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 37 v 25 (ObjNum 59 snap 1 seq_num 59) dirty exists 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: left oid 37 (ObjNum 59 snap 1 seq_num 59) 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:79: done (3 left) 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:80: done (2 left) 2024-09-16T04:01:22.876 INFO:tasks.rados.rados.0.smithi188.stdout:81: done (1 left) 2024-09-16T04:01:22.877 INFO:tasks.rados.rados.0.smithi188.stdout:83: rmattr oid 2 current snap is 1 2024-09-16T04:01:22.877 INFO:tasks.rados.rados.0.smithi188.stdout:83: done (1 left) 2024-09-16T04:01:22.877 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 46 current snap is 1 2024-09-16T04:01:22.877 INFO:tasks.rados.rados.0.smithi188.stdout:84: seq_num 60 ranges {770048=638976,1409024=376832} 2024-09-16T04:01:22.894 INFO:tasks.rados.rados.0.smithi188.stdout:84: writing smithi18831615-46 from 770048 to 1409024 tid 1 2024-09-16T04:01:22.899 INFO:tasks.rados.rados.0.smithi188.stdout:84: writing smithi18831615-46 from 1409024 to 1785856 tid 2 2024-09-16T04:01:22.899 INFO:tasks.rados.rados.0.smithi188.stdout:85: setattr oid 40 current snap is 1 2024-09-16T04:01:22.899 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T04:01:22.901 INFO:tasks.rados.rados.0.smithi188.stdout:82: done (2 left) 2024-09-16T04:01:22.901 INFO:tasks.rados.rados.0.smithi188.stdout:86: delete oid 39 current snap is 1 2024-09-16T04:01:22.905 INFO:tasks.rados.rados.0.smithi188.stdout:86: done (2 left) 2024-09-16T04:01:22.906 INFO:tasks.rados.rados.0.smithi188.stdout:87: delete oid 17 current snap is 1 2024-09-16T04:01:22.913 INFO:tasks.rados.rados.0.smithi188.stdout:84: finishing write tid 1 to smithi18831615-46 2024-09-16T04:01:22.915 INFO:tasks.rados.rados.0.smithi188.stdout:87: done (2 left) 2024-09-16T04:01:22.916 INFO:tasks.rados.rados.0.smithi188.stdout:88: read oid 42 snap 0 2024-09-16T04:01:22.916 INFO:tasks.rados.rados.0.smithi188.stdout:88: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:22.916 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 17 current snap is 1 2024-09-16T04:01:22.916 INFO:tasks.rados.rados.0.smithi188.stdout:89: seq_num 63 ranges {0=688128,688128=376832} 2024-09-16T04:01:22.924 INFO:tasks.rados.rados.0.smithi188.stdout:89: writing smithi18831615-17 from 0 to 688128 tid 1 2024-09-16T04:01:22.927 INFO:tasks.rados.rados.0.smithi188.stdout:89: writing smithi18831615-17 from 688128 to 1064960 tid 2 2024-09-16T04:01:22.928 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 47 current snap is 1 2024-09-16T04:01:22.928 INFO:tasks.rados.rados.0.smithi188.stdout:90: seq_num 64 ranges {786432=704512,1490944=376832} 2024-09-16T04:01:22.936 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:22.936 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:22.943 INFO:tasks.rados.rados.0.smithi188.stdout:90: writing smithi18831615-47 from 786432 to 1490944 tid 1 2024-09-16T04:01:22.947 INFO:tasks.rados.rados.0.smithi188.stdout:90: writing smithi18831615-47 from 1490944 to 1867776 tid 2 2024-09-16T04:01:22.947 INFO:tasks.rados.rados.0.smithi188.stdout:91: rollback oid 38 current snap is 1 2024-09-16T04:01:22.947 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 38 to 0 2024-09-16T04:01:22.947 INFO:tasks.rados.rados.0.smithi188.stdout:84: finishing write tid 2 to smithi18831615-46 2024-09-16T04:01:22.947 INFO:tasks.rados.rados.0.smithi188.stdout:92: copy_from oid 34 from oid 43 current snap is 1 2024-09-16T04:01:22.947 INFO:tasks.rados.rados.0.smithi188.stdout:84: finishing write tid 3 to smithi18831615-46 2024-09-16T04:01:22.947 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 40 v 18 (ObjNum 33 snap 0 seq_num 3904101296) dirty exists 2024-09-16T04:01:22.948 INFO:tasks.rados.rados.0.smithi188.stdout:84: finishing write tid 4 to smithi18831615-46 2024-09-16T04:01:22.948 INFO:tasks.rados.rados.0.smithi188.stdout:84: oid 46 updating version 0 to 17 2024-09-16T04:01:22.948 INFO:tasks.rados.rados.0.smithi188.stdout:84: oid 46 version 17 is already newer than 16 2024-09-16T04:01:22.948 INFO:tasks.rados.rados.0.smithi188.stdout:84: oid 46 version 17 is already newer than 15 2024-09-16T04:01:22.948 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 46 v 17 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-09-16T04:01:22.948 INFO:tasks.rados.rados.0.smithi188.stdout:84: left oid 46 (ObjNum 60 snap 1 seq_num 60) 2024-09-16T04:01:22.948 INFO:tasks.rados.rados.0.smithi188.stdout:88: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:22.956 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:01:22.965 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:22.961+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:89: finishing write tid 1 to smithi18831615-17 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:89: finishing write tid 2 to smithi18831615-17 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:89: finishing write tid 3 to smithi18831615-17 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:89: finishing write tid 4 to smithi18831615-17 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:89: oid 17 updating version 0 to 8 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:89: oid 17 updating version 8 to 9 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:89: oid 17 version 9 is already newer than 7 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 17 v 9 (ObjNum 63 snap 1 seq_num 63) dirty exists 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:89: left oid 17 (ObjNum 63 snap 1 seq_num 63) 2024-09-16T04:01:22.981 INFO:tasks.rados.rados.0.smithi188.stdout:84: done (6 left) 2024-09-16T04:01:22.982 INFO:tasks.rados.rados.0.smithi188.stdout:85: done (5 left) 2024-09-16T04:01:22.982 INFO:tasks.rados.rados.0.smithi188.stdout:88: done (4 left) 2024-09-16T04:01:22.982 INFO:tasks.rados.rados.0.smithi188.stdout:89: done (3 left) 2024-09-16T04:01:22.982 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 27 current snap is 1 2024-09-16T04:01:22.982 INFO:tasks.rados.rados.0.smithi188.stdout:93: seq_num 65 ranges {442368=720896,1163264=376832} 2024-09-16T04:01:22.987 INFO:tasks.rados.rados.0.smithi188.stdout:93: writing smithi18831615-27 from 442368 to 1163264 tid 1 2024-09-16T04:01:22.989 INFO:tasks.rados.rados.0.smithi188.stdout:93: writing smithi18831615-27 from 1163264 to 1540096 tid 2 2024-09-16T04:01:22.989 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 35 current snap is 1 2024-09-16T04:01:22.989 INFO:tasks.rados.rados.0.smithi188.stdout:94: seq_num 66 ranges {573440=737280,1310720=376832} 2024-09-16T04:01:22.996 INFO:tasks.rados.rados.0.smithi188.stdout:94: writing smithi18831615-35 from 573440 to 1310720 tid 1 2024-09-16T04:01:22.999 INFO:tasks.rados.rados.0.smithi188.stdout:94: writing smithi18831615-35 from 1310720 to 1687552 tid 2 2024-09-16T04:01:22.999 INFO:tasks.rados.rados.0.smithi188.stdout:95: rmattr oid 19 current snap is 1 2024-09-16T04:01:22.999 INFO:tasks.rados.rados.0.smithi188.stdout:95: done (5 left) 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:96: delete oid 19 current snap is 1 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:90: finishing write tid 1 to smithi18831615-47 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:91: finishing rollback tid 0 to smithi18831615-38 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:91: finishing rollback tid 1 to smithi18831615-38 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:90: finishing write tid 2 to smithi18831615-47 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:91: finishing rollback tid 2 to smithi18831615-38 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 38 v 29 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:90: finishing write tid 3 to smithi18831615-47 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:90: finishing write tid 4 to smithi18831615-47 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:90: oid 47 updating version 0 to 11 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:90: oid 47 version 11 is already newer than 9 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:90: oid 47 version 11 is already newer than 10 2024-09-16T04:01:23.000 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 47 v 11 (ObjNum 64 snap 1 seq_num 64) dirty exists 2024-09-16T04:01:23.001 INFO:tasks.rados.rados.0.smithi188.stdout:90: left oid 47 (ObjNum 64 snap 1 seq_num 64) 2024-09-16T04:01:23.001 INFO:tasks.rados.rados.0.smithi188.stdout:92: finishing copy_from racing read to smithi18831615-34 2024-09-16T04:01:23.001 INFO:tasks.rados.rados.0.smithi188.stdout:92: finishing copy_from to smithi18831615-34 2024-09-16T04:01:23.001 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 34 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T04:01:23.001 INFO:tasks.rados.rados.0.smithi188.stdout:93: finishing write tid 1 to smithi18831615-27 2024-09-16T04:01:23.003 INFO:tasks.rados.rados.0.smithi188.stdout:93: finishing write tid 2 to smithi18831615-27 2024-09-16T04:01:23.003 INFO:tasks.rados.rados.0.smithi188.stdout:93: finishing write tid 3 to smithi18831615-27 2024-09-16T04:01:23.005 INFO:tasks.rados.rados.0.smithi188.stdout:94: finishing write tid 1 to smithi18831615-35 2024-09-16T04:01:23.006 INFO:tasks.rados.rados.0.smithi188.stdout:94: finishing write tid 2 to smithi18831615-35 2024-09-16T04:01:23.006 INFO:tasks.rados.rados.0.smithi188.stdout:94: finishing write tid 3 to smithi18831615-35 2024-09-16T04:01:23.006 INFO:tasks.rados.rados.0.smithi188.stdout:90: done (5 left) 2024-09-16T04:01:23.006 INFO:tasks.rados.rados.0.smithi188.stdout:91: done (4 left) 2024-09-16T04:01:23.007 INFO:tasks.rados.rados.0.smithi188.stdout:92: done (3 left) 2024-09-16T04:01:23.007 INFO:tasks.rados.rados.0.smithi188.stdout:96: done (2 left) 2024-09-16T04:01:23.007 INFO:tasks.rados.rados.0.smithi188.stdout:97: setattr oid 8 current snap is 1 2024-09-16T04:01:23.007 INFO:tasks.rados.rados.0.smithi188.stdout:93: finishing write tid 4 to smithi18831615-27 2024-09-16T04:01:23.007 INFO:tasks.rados.rados.0.smithi188.stdout:93: oid 27 updating version 0 to 5 2024-09-16T04:01:23.007 INFO:tasks.rados.rados.0.smithi188.stdout:93: oid 27 updating version 5 to 6 2024-09-16T04:01:23.007 INFO:tasks.rados.rados.0.smithi188.stdout:93: oid 27 version 6 is already newer than 4 2024-09-16T04:01:23.008 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 27 v 6 (ObjNum 65 snap 1 seq_num 65) dirty exists 2024-09-16T04:01:23.008 INFO:tasks.rados.rados.0.smithi188.stdout:93: left oid 27 (ObjNum 65 snap 1 seq_num 65) 2024-09-16T04:01:23.008 INFO:tasks.rados.rados.0.smithi188.stdout:94: finishing write tid 4 to smithi18831615-35 2024-09-16T04:01:23.008 INFO:tasks.rados.rados.0.smithi188.stdout:94: oid 35 updating version 0 to 13 2024-09-16T04:01:23.009 INFO:tasks.rados.rados.0.smithi188.stdout:94: oid 35 updating version 13 to 14 2024-09-16T04:01:23.009 INFO:tasks.rados.rados.0.smithi188.stdout:94: oid 35 updating version 14 to 15 2024-09-16T04:01:23.009 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 15 (ObjNum 66 snap 1 seq_num 66) dirty exists 2024-09-16T04:01:23.009 INFO:tasks.rados.rados.0.smithi188.stdout:94: left oid 35 (ObjNum 66 snap 1 seq_num 66) 2024-09-16T04:01:23.009 INFO:tasks.rados.rados.0.smithi188.stdout:93: done (2 left) 2024-09-16T04:01:23.009 INFO:tasks.rados.rados.0.smithi188.stdout:94: done (1 left) 2024-09-16T04:01:23.009 INFO:tasks.rados.rados.0.smithi188.stdout:98: setattr oid 21 current snap is 1 2024-09-16T04:01:23.010 INFO:tasks.rados.rados.0.smithi188.stdout:99: rollback oid 24 current snap is 1 2024-09-16T04:01:23.010 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 24 to 0 2024-09-16T04:01:23.010 INFO:tasks.rados.rados.0.smithi188.stdout:100: rmattr oid 15 current snap is 1 2024-09-16T04:01:23.011 INFO:tasks.rados.rados.0.smithi188.stdout:100: done (3 left) 2024-09-16T04:01:23.011 INFO:tasks.rados.rados.0.smithi188.stdout:101: read oid 23 snap -1 2024-09-16T04:01:23.011 INFO:tasks.rados.rados.0.smithi188.stdout:101: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:01:23.011 INFO:tasks.rados.rados.0.smithi188.stdout:102: read oid 19 snap -1 2024-09-16T04:01:23.011 INFO:tasks.rados.rados.0.smithi188.stdout:102: expect deleted 2024-09-16T04:01:23.011 INFO:tasks.rados.rados.0.smithi188.stdout:103: read oid 35 snap 0 2024-09-16T04:01:23.011 INFO:tasks.rados.rados.0.smithi188.stdout:103: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T04:01:23.011 INFO:tasks.rados.rados.0.smithi188.stdout:104: snap_create 2024-09-16T04:01:23.017 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 8 v 14 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T04:01:23.020 INFO:tasks.rados.rados.0.smithi188.stdout:101: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:01:23.021 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:23.014+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:23.030 INFO:tasks.rados.rados.0.smithi188.stdout:103: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T04:01:23.045 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 21 v 31 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T04:01:23.045 INFO:tasks.rados.rados.0.smithi188.stdout:99: finishing rollback tid 0 to smithi18831615-24 2024-09-16T04:01:23.045 INFO:tasks.rados.rados.0.smithi188.stdout:99: finishing rollback tid 1 to smithi18831615-24 2024-09-16T04:01:23.045 INFO:tasks.rados.rados.0.smithi188.stdout:99: finishing rollback tid 2 to smithi18831615-24 2024-09-16T04:01:23.045 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 24 v 35 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T04:01:23.107 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:23.108 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:23.122 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:01:23.122 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:23.118+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:23.222 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:23.218+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:23.286 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:23.287 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:23.300 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:01:23.323 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:23.318+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:23.425 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:23.422+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:23.443 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:01:23.526 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:23.522+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:23.586 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:01:23.626 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:23.622+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:23.728 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:23.721+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:23.729 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:01:23.829 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:23.826+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:23.858 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T04:01:23.865 INFO:tasks.rados.rados.0.smithi188.stdout:97: done (6 left) 2024-09-16T04:01:23.865 INFO:tasks.rados.rados.0.smithi188.stdout:98: done (5 left) 2024-09-16T04:01:23.865 INFO:tasks.rados.rados.0.smithi188.stdout:99: done (4 left) 2024-09-16T04:01:23.865 INFO:tasks.rados.rados.0.smithi188.stdout:101: done (3 left) 2024-09-16T04:01:23.865 INFO:tasks.rados.rados.0.smithi188.stdout:102: done (2 left) 2024-09-16T04:01:23.865 INFO:tasks.rados.rados.0.smithi188.stdout:103: done (1 left) 2024-09-16T04:01:23.865 INFO:tasks.rados.rados.0.smithi188.stdout:104: done (0 left) 2024-09-16T04:01:23.865 INFO:tasks.rados.rados.0.smithi188.stdout:105: rollback oid 42 current snap is 2 2024-09-16T04:01:23.866 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 42 to 1 2024-09-16T04:01:23.866 INFO:tasks.rados.rados.0.smithi188.stdout:106: read oid 32 snap -1 2024-09-16T04:01:23.866 INFO:tasks.rados.rados.0.smithi188.stdout:106: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T04:01:23.866 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 3 current snap is 2 2024-09-16T04:01:23.866 INFO:tasks.rados.rados.0.smithi188.stdout:107: seq_num 68 ranges {688128=770048,1458176=376832} 2024-09-16T04:01:23.881 INFO:tasks.rados.rados.0.smithi188.stdout:107: writing smithi18831615-3 from 688128 to 1458176 tid 1 2024-09-16T04:01:23.885 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T04:01:23.886 INFO:tasks.rados.rados.0.smithi188.stdout:107: writing smithi18831615-3 from 1458176 to 1835008 tid 2 2024-09-16T04:01:23.886 INFO:tasks.rados.rados.0.smithi188.stdout:108: setattr oid 35 current snap is 2 2024-09-16T04:01:23.886 INFO:tasks.rados.rados.0.smithi188.stdout:106: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T04:01:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:23.908 INFO:tasks.rados.rados.0.smithi188.stdout:105: finishing rollback tid 0 to smithi18831615-42 2024-09-16T04:01:23.908 INFO:tasks.rados.rados.0.smithi188.stdout:105: finishing rollback tid 1 to smithi18831615-42 2024-09-16T04:01:23.908 INFO:tasks.rados.rados.0.smithi188.stdout:105: finishing rollback tid 2 to smithi18831615-42 2024-09-16T04:01:23.908 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T04:01:23.908 INFO:tasks.rados.rados.0.smithi188.stdout:107: finishing write tid 1 to smithi18831615-3 2024-09-16T04:01:23.908 INFO:tasks.rados.rados.0.smithi188.stdout:107: finishing write tid 2 to smithi18831615-3 2024-09-16T04:01:23.908 INFO:tasks.rados.rados.0.smithi188.stdout:107: finishing write tid 3 to smithi18831615-3 2024-09-16T04:01:23.908 INFO:tasks.rados.rados.0.smithi188.stdout:107: finishing write tid 4 to smithi18831615-3 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:107: oid 3 updating version 0 to 22 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:107: oid 3 version 22 is already newer than 21 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:107: oid 3 version 22 is already newer than 20 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 3 v 22 (ObjNum 68 snap 2 seq_num 68) dirty exists 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:107: left oid 3 (ObjNum 68 snap 2 seq_num 68) 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:105: done (3 left) 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:106: done (2 left) 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:107: done (1 left) 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:109: copy_from oid 10 from oid 23 current snap is 2 2024-09-16T04:01:23.909 INFO:tasks.rados.rados.0.smithi188.stdout:110: rollback oid 24 current snap is 2 2024-09-16T04:01:23.910 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 24 to 0 2024-09-16T04:01:23.910 INFO:tasks.rados.rados.0.smithi188.stdout:111: rmattr oid 9 current snap is 2 2024-09-16T04:01:23.910 INFO:tasks.rados.rados.0.smithi188.stdout:111: done (3 left) 2024-09-16T04:01:23.910 INFO:tasks.rados.rados.0.smithi188.stdout:112: rmattr oid 27 current snap is 2 2024-09-16T04:01:23.910 INFO:tasks.rados.rados.0.smithi188.stdout:113: read oid 16 snap -1 2024-09-16T04:01:23.910 INFO:tasks.rados.rados.0.smithi188.stdout:113: expect (ObjNum 54 snap 0 seq_num 54) 2024-09-16T04:01:23.910 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 44 current snap is 2 2024-09-16T04:01:23.910 INFO:tasks.rados.rados.0.smithi188.stdout:114: seq_num 69 ranges {737280=786432,1523712=376832} 2024-09-16T04:01:23.921 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:01:23.925 INFO:tasks.rados.rados.0.smithi188.stdout:114: writing smithi18831615-44 from 737280 to 1523712 tid 1 2024-09-16T04:01:23.929 INFO:tasks.rados.rados.0.smithi188.stdout:114: writing smithi18831615-44 from 1523712 to 1900544 tid 2 2024-09-16T04:01:23.929 INFO:tasks.rados.rados.0.smithi188.stdout:115: snap_create 2024-09-16T04:01:23.929 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 27 v 0 (ObjNum 65 snap 1 seq_num 65) dirty exists 2024-09-16T04:01:23.929 INFO:tasks.rados.rados.0.smithi188.stdout:110: finishing rollback tid 0 to smithi18831615-24 2024-09-16T04:01:23.930 INFO:tasks.rados.rados.0.smithi188.stdout:110: finishing rollback tid 1 to smithi18831615-24 2024-09-16T04:01:23.930 INFO:tasks.rados.rados.0.smithi188.stdout:110: finishing rollback tid 2 to smithi18831615-24 2024-09-16T04:01:23.930 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 24 v 39 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T04:01:23.930 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 16 (ObjNum 66 snap 1 seq_num 66) dirty exists 2024-09-16T04:01:23.930 INFO:tasks.rados.rados.0.smithi188.stdout:113: expect (ObjNum 54 snap 0 seq_num 54) 2024-09-16T04:01:23.965 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:23.961+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:23.984 INFO:tasks.rados.rados.0.smithi188.stdout:109: finishing copy_from racing read to smithi18831615-10 2024-09-16T04:01:23.985 INFO:tasks.rados.rados.0.smithi188.stdout:109: finishing copy_from to smithi18831615-10 2024-09-16T04:01:23.985 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 10 v 2 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T04:01:23.985 INFO:tasks.rados.rados.0.smithi188.stdout:114: finishing write tid 1 to smithi18831615-44 2024-09-16T04:01:23.985 INFO:tasks.rados.rados.0.smithi188.stdout:114: finishing write tid 2 to smithi18831615-44 2024-09-16T04:01:23.985 INFO:tasks.rados.rados.0.smithi188.stdout:114: finishing write tid 3 to smithi18831615-44 2024-09-16T04:01:23.986 INFO:tasks.rados.rados.0.smithi188.stdout:114: finishing write tid 4 to smithi18831615-44 2024-09-16T04:01:23.986 INFO:tasks.rados.rados.0.smithi188.stdout:114: oid 44 updating version 0 to 9 2024-09-16T04:01:23.986 INFO:tasks.rados.rados.0.smithi188.stdout:114: oid 44 updating version 9 to 10 2024-09-16T04:01:23.986 INFO:tasks.rados.rados.0.smithi188.stdout:114: oid 44 updating version 10 to 11 2024-09-16T04:01:23.986 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 44 v 11 (ObjNum 69 snap 2 seq_num 69) dirty exists 2024-09-16T04:01:23.986 INFO:tasks.rados.rados.0.smithi188.stdout:114: left oid 44 (ObjNum 69 snap 2 seq_num 69) 2024-09-16T04:01:24.031 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:24.026+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:24.099 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:01:24.108 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:01:24.131 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:24.126+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:24.233 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:01:24.234 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:24.226+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:24.300 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:24.321 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:24.333 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:24.326+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:24.342 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:24.362 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:24.367 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:01:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:24.380 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:24.401 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:24.422 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:24.434 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:24.429+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:24.444 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:24.463 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:24.487 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:24.487 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:24.487 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:24.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:01:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:24.531 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:24.536 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:24.534+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:24.550 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:24.569+0000 7f3200de6640 0 -- 172.21.15.37:0/3298429880 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7f31fc227e30 msgr2=0x7f31fc22a220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:24.621 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:24.624 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:24.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:24.624 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [] 2024-09-16T04:01:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:24.637 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:24.634+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:24.641 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:01:24.738 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:24.734+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:24.813 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:24.813 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:24.829 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:01:24.839 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:24.834+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:24.861 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:108: done (6 left) 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:109: done (5 left) 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:110: done (4 left) 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:112: done (3 left) 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:113: done (2 left) 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:114: done (1 left) 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:115: done (0 left) 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 24 current snap is 3 2024-09-16T04:01:24.869 INFO:tasks.rados.rados.0.smithi188.stdout:116: seq_num 70 ranges {393216=802816,1196032=376832} 2024-09-16T04:01:24.874 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:01:24.883 INFO:tasks.rados.rados.0.smithi188.stdout:116: writing smithi18831615-24 from 393216 to 1196032 tid 1 2024-09-16T04:01:24.889 INFO:tasks.rados.rados.0.smithi188.stdout:116: writing smithi18831615-24 from 1196032 to 1572864 tid 2 2024-09-16T04:01:24.889 INFO:tasks.rados.rados.0.smithi188.stdout:117: snap_remove snap 2 2024-09-16T04:01:24.891 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T04:01:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:24.963 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:24.963 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:24.963 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:24.964 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:24.957+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:24.979 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:01:25.041 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:25.038+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:25.141 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:25.137+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:25.144 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:25.144 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:25.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:25.145 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:25.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.145 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_repop(client.4371.0:5 2.2 e29/27 2:5cea7035:::rbd_info:head v 29'1, mlcod=0'0)", 2024-09-16T04:01:25.145 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:07.739226+0000", 2024-09-16T04:01:25.146 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.403337718, 2024-09-16T04:01:25.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001906778, 2024-09-16T04:01:25.146 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:25.146 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:25.146 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:25.147 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:25.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:25.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.739226+0000", 2024-09-16T04:01:25.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:25.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.739226+0000", 2024-09-16T04:01:25.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:25.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.739227+0000", 2024-09-16T04:01:25.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8699999999999995e-07 2024-09-16T04:01:25.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:25.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.739262+0000", 2024-09-16T04:01:25.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5005000000000001e-05 2024-09-16T04:01:25.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:25.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.739286+0000", 2024-09-16T04:01:25.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4596e-05 2024-09-16T04:01:25.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:25.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.739302+0000", 2024-09-16T04:01:25.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5970000000000001e-05 2024-09-16T04:01:25.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:25.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.739485+0000", 2024-09-16T04:01:25.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018254799999999999 2024-09-16T04:01:25.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:25.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.739532+0000", 2024-09-16T04:01:25.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7487999999999997e-05 2024-09-16T04:01:25.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.155 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:25.155 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.741042+0000", 2024-09-16T04:01:25.155 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001510157 2024-09-16T04:01:25.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:25.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:07.741133+0000", 2024-09-16T04:01:25.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0426999999999993e-05 2024-09-16T04:01:25.156 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.156 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:25.157 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.157 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_repop(client.4371.0:7 2.2 e30/27 2:5cea7035:::rbd_info:head v 30'2, mlcod=0'0)", 2024-09-16T04:01:25.157 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:08.744414+0000", 2024-09-16T04:01:25.158 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.398149873000001, 2024-09-16T04:01:25.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0019670349999999998, 2024-09-16T04:01:25.158 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:25.158 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:25.158 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:25.158 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:25.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.159 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:25.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.744414+0000", 2024-09-16T04:01:25.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.160 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:25.160 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.744414+0000", 2024-09-16T04:01:25.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:25.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.744414+0000", 2024-09-16T04:01:25.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5700000000000002e-07 2024-09-16T04:01:25.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:25.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.744423+0000", 2024-09-16T04:01:25.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3729999999999993e-06 2024-09-16T04:01:25.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.162 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:25.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.744451+0000", 2024-09-16T04:01:25.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7898999999999998e-05 2024-09-16T04:01:25.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:25.163 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.744468+0000", 2024-09-16T04:01:25.163 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7535999999999999e-05 2024-09-16T04:01:25.163 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:25.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.744627+0000", 2024-09-16T04:01:25.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015876900000000001 2024-09-16T04:01:25.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.164 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:25.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.744666+0000", 2024-09-16T04:01:25.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9152000000000001e-05 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.746292+0000", 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016262850000000001 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:08.746381+0000", 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8463999999999999e-05 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:25.165 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.25399640800000001, 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0044144730000000004, 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:25.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888568+0000", 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.67e-07 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:25.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890276+0000", 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0017087719999999999 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890297+0000", 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0384e-05 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890309+0000", 2024-09-16T04:01:25.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1827999999999999e-05 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890402+0000", 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3830000000000001e-05 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890412+0000", 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8050000000000001e-06 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892961+0000", 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0025483979999999999 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892982+0000", 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1189e-05 2024-09-16T04:01:25.170 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 39/33 ECSubWrite(tid=81, reqid=client.4646.0:292, at_version=39'42, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.891848+0000", 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.250715998, 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003440556, 2024-09-16T04:01:25.171 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891848+0000", 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891848+0000", 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891849+0000", 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9400000000000004e-07 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891935+0000", 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6601999999999995e-05 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:25.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891949+0000", 2024-09-16T04:01:25.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3878e-05 2024-09-16T04:01:25.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891959+0000", 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7399999999999999e-06 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892017+0000", 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8057999999999999e-05 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892026+0000", 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4630000000000007e-06 2024-09-16T04:01:25.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.895246+0000", 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003219965 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.895288+0000", 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1956000000000003e-05 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:25.176 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 39/33 ECSubWrite(tid=82, reqid=client.4646.0:293, at_version=39'43, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.893728+0000", 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.24883540700000001, 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021113820000000002, 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893728+0000", 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893728+0000", 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:25.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893729+0000", 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4900000000000005e-07 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893748+0000", 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8595999999999999e-05 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893764+0000", 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6067999999999999e-05 2024-09-16T04:01:25.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.180 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:25.180 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893773+0000", 2024-09-16T04:01:25.180 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8570000000000001e-06 2024-09-16T04:01:25.180 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.181 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:25.181 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893867+0000", 2024-09-16T04:01:25.181 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4529000000000002e-05 2024-09-16T04:01:25.181 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.181 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.181 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:25.181 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893876+0000", 2024-09-16T04:01:25.181 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5739999999999996e-06 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.895813+0000", 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001937287 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.895840+0000", 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6721999999999999e-05 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:25.182 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 39/33 ECSubWrite(tid=83, reqid=client.4646.0:293, at_version=0'0, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.897851+0000", 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.24471278699999999, 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00072034599999999996, 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.897851+0000", 2024-09-16T04:01:25.183 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.897851+0000", 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.897852+0000", 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2999999999999999e-07 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.897867+0000", 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5274e-05 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.897885+0000", 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7782999999999999e-05 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.897894+0000", 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9309999999999997e-06 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.897983+0000", 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9019999999999998e-05 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.897993+0000", 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0431999999999999e-05 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.898536+0000", 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00054317999999999999 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.898571+0000", 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4895999999999999e-05 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:25.187 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:25.189 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:01:25.242 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:25.238+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:25.323 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:25.323 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:25.323 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:25.323 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:25.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:01:25.343 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:25.342+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:25.444 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:25.442+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:25.464 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:25.464 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:25.464 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:25.464 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:25.464 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:25.478 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:01:25.545 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:25.538+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:25.605 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [] 2024-09-16T04:01:25.606 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:25.624 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:01:25.666 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:25.662+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:25.747 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:25.742+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:25.793 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:25.794 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:25.794 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:25.794 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:25.807 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:01:25.848 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:25.846+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:25.882 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-09-16T04:01:25.882 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T04:01:25.887 INFO:tasks.rados.rados.0.smithi188.stdout:117: done (1 left) 2024-09-16T04:01:25.887 INFO:tasks.rados.rados.0.smithi188.stdout:118: rollback oid 31 current snap is 3 2024-09-16T04:01:25.887 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 31 to 1 2024-09-16T04:01:25.887 INFO:tasks.rados.rados.0.smithi188.stdout:116: finishing write tid 1 to smithi18831615-24 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:116: finishing write tid 2 to smithi18831615-24 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:116: finishing write tid 3 to smithi18831615-24 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:116: finishing write tid 4 to smithi18831615-24 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:116: oid 24 updating version 0 to 42 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:116: oid 24 version 42 is already newer than 41 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:116: oid 24 updating version 42 to 43 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 24 v 43 (ObjNum 70 snap 3 seq_num 70) dirty exists 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:116: left oid 24 (ObjNum 70 snap 3 seq_num 70) 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:116: done (1 left) 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 21 current snap is 3 2024-09-16T04:01:25.888 INFO:tasks.rados.rados.0.smithi188.stdout:119: seq_num 71 ranges {344064=425984,770048=573440,1343488=196608} 2024-09-16T04:01:25.899 INFO:tasks.rados.rados.0.smithi188.stdout:119: writing smithi18831615-21 from 344064 to 770048 tid 1 2024-09-16T04:01:25.907 INFO:tasks.rados.rados.0.smithi188.stdout:119: writing smithi18831615-21 from 770048 to 1343488 tid 2 2024-09-16T04:01:25.911 INFO:tasks.rados.rados.0.smithi188.stdout:119: writing smithi18831615-21 from 1343488 to 1540096 tid 3 2024-09-16T04:01:25.911 INFO:tasks.rados.rados.0.smithi188.stdout:120: setattr oid 32 current snap is 3 2024-09-16T04:01:25.911 INFO:tasks.rados.rados.0.smithi188.stdout:118: finishing rollback tid 1 to smithi18831615-31 2024-09-16T04:01:25.911 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 31 v 0 (ObjNum 33 snap 0 seq_num 2283926128) dirty dne 2024-09-16T04:01:25.913 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T04:01:25.918 INFO:tasks.rados.rados.0.smithi188.stdout:118: done (2 left) 2024-09-16T04:01:25.918 INFO:tasks.rados.rados.0.smithi188.stdout:121: delete oid 4 current snap is 3 2024-09-16T04:01:25.919 INFO:tasks.rados.rados.0.smithi188.stdout:119: finishing write tid 1 to smithi18831615-21 2024-09-16T04:01:25.921 INFO:tasks.rados.rados.0.smithi188.stdout:119: finishing write tid 2 to smithi18831615-21 2024-09-16T04:01:25.924 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 32 v 8 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T04:01:25.927 INFO:tasks.rados.rados.0.smithi188.stdout:119: finishing write tid 3 to smithi18831615-21 2024-09-16T04:01:25.930 INFO:tasks.rados.rados.0.smithi188.stdout:119: finishing write tid 4 to smithi18831615-21 2024-09-16T04:01:25.933 INFO:tasks.rados.rados.0.smithi188.stdout:120: done (2 left) 2024-09-16T04:01:25.933 INFO:tasks.rados.rados.0.smithi188.stdout:121: done (1 left) 2024-09-16T04:01:25.933 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 3 current snap is 3 2024-09-16T04:01:25.933 INFO:tasks.rados.rados.0.smithi188.stdout:122: seq_num 73 ranges {1835008=458752,2293760=475136,2768896=294912} 2024-09-16T04:01:25.949 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:25.942+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:25.964 INFO:tasks.rados.rados.0.smithi188.stdout:122: writing smithi18831615-3 from 1835008 to 2293760 tid 1 2024-09-16T04:01:25.970 INFO:tasks.rados.rados.0.smithi188.stdout:122: writing smithi18831615-3 from 2293760 to 2768896 tid 2 2024-09-16T04:01:25.974 INFO:tasks.rados.rados.0.smithi188.stdout:122: writing smithi18831615-3 from 2768896 to 3063808 tid 3 2024-09-16T04:01:25.974 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 50 current snap is 3 2024-09-16T04:01:25.975 INFO:tasks.rados.rados.0.smithi188.stdout:123: seq_num 74 ranges {835584=475136,1310720=622592,1933312=147456} 2024-09-16T04:01:25.987 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:25.987 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:25.987 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:25.987 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:25.987 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:25.987 INFO:tasks.rados.rados.0.smithi188.stdout:123: writing smithi18831615-50 from 835584 to 1310720 tid 1 2024-09-16T04:01:25.993 INFO:tasks.rados.rados.0.smithi188.stdout:123: writing smithi18831615-50 from 1310720 to 1933312 tid 2 2024-09-16T04:01:25.994 INFO:tasks.rados.rados.0.smithi188.stdout:123: writing smithi18831615-50 from 1933312 to 2080768 tid 3 2024-09-16T04:01:25.995 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 36 current snap is 3 2024-09-16T04:01:25.995 INFO:tasks.rados.rados.0.smithi188.stdout:124: seq_num 75 ranges {1490944=491520,1982464=770048} 2024-09-16T04:01:26.003 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:01:26.010 INFO:tasks.rados.rados.0.smithi188.stdout:124: writing smithi18831615-36 from 1490944 to 1982464 tid 1 2024-09-16T04:01:26.015 INFO:tasks.rados.rados.0.smithi188.stdout:124: writing smithi18831615-36 from 1982464 to 2752512 tid 2 2024-09-16T04:01:26.015 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 17 current snap is 3 2024-09-16T04:01:26.016 INFO:tasks.rados.rados.0.smithi188.stdout:125: seq_num 76 ranges {1064960=507904,1572864=507904,2080768=262144} 2024-09-16T04:01:26.025 INFO:tasks.rados.rados.0.smithi188.stdout:125: writing smithi18831615-17 from 1064960 to 1572864 tid 1 2024-09-16T04:01:26.029 INFO:tasks.rados.rados.0.smithi188.stdout:125: writing smithi18831615-17 from 1572864 to 2080768 tid 2 2024-09-16T04:01:26.031 INFO:tasks.rados.rados.0.smithi188.stdout:125: writing smithi18831615-17 from 2080768 to 2342912 tid 3 2024-09-16T04:01:26.031 INFO:tasks.rados.rados.0.smithi188.stdout:126: rollback oid 49 current snap is 3 2024-09-16T04:01:26.031 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 49 to 1 2024-09-16T04:01:26.031 INFO:tasks.rados.rados.0.smithi188.stdout:119: finishing write tid 5 to smithi18831615-21 2024-09-16T04:01:26.031 INFO:tasks.rados.rados.0.smithi188.stdout:119: oid 21 updating version 0 to 49 2024-09-16T04:01:26.031 INFO:tasks.rados.rados.0.smithi188.stdout:119: oid 21 version 49 is already newer than 45 2024-09-16T04:01:26.031 INFO:tasks.rados.rados.0.smithi188.stdout:119: oid 21 version 49 is already newer than 48 2024-09-16T04:01:26.031 INFO:tasks.rados.rados.0.smithi188.stdout:119: oid 21 updating version 49 to 50 2024-09-16T04:01:26.032 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 21 v 50 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-09-16T04:01:26.032 INFO:tasks.rados.rados.0.smithi188.stdout:119: left oid 21 (ObjNum 71 snap 3 seq_num 71) 2024-09-16T04:01:26.032 INFO:tasks.rados.rados.0.smithi188.stdout:119: done (5 left) 2024-09-16T04:01:26.032 INFO:tasks.rados.rados.0.smithi188.stdout:127: snap_remove snap 0 2024-09-16T04:01:26.050 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:26.046+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:26.130 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:26.130 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:26.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:26.130 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:26.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.130 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4371.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:01:26.130 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:01:26.130 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.411990481, 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01704027, 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4371", 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.37:0/2503621667", 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:01:26.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716696+0000", 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:26.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716711+0000", 2024-09-16T04:01:26.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4676e-05 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716743+0000", 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1835000000000003e-05 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716769+0000", 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6327999999999999e-05 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716973+0000", 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020410900000000001 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.717006+0000", 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3084999999999999e-05 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.724942+0000", 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0079355829999999995 2024-09-16T04:01:26.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.724964+0000", 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1804999999999999e-05 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730286+0000", 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053221689999999999 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:26.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730641+0000", 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035484999999999998 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730957+0000", 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0003165 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.732250+0000", 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001293039 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733535+0000", 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001284477 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733616+0000", 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0964999999999995e-05 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:26.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733727+0000", 2024-09-16T04:01:26.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011134 2024-09-16T04:01:26.140 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.140 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.140 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop_reply(client.4371.0:1 2.3 e28/27 ondisk, result = 0)", 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:06.733354+0000", 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.395323149999999, 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000386492, 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:26.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733354+0000", 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733354+0000", 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:26.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733356+0000", 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.68e-06 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733366+0000", 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8230000000000006e-06 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733392+0000", 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6658000000000001e-05 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733409+0000", 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6659000000000001e-05 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733511+0000", 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000101891 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:26.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733526+0000", 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5322999999999999e-05 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733741+0000", 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000214458 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:24.888075+0000", 2024-09-16T04:01:26.146 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.2406024389999999, 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052425479999999997, 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888075+0000", 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888075+0000", 2024-09-16T04:01:26.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888075+0000", 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.84e-07 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888746+0000", 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067078999999999999 2024-09-16T04:01:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888787+0000", 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0757999999999998e-05 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888807+0000", 2024-09-16T04:01:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0775999999999999e-05 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888982+0000", 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000174277 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.889009+0000", 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6923e-05 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.893256+0000", 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042475059999999999 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.893317+0000", 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1134000000000001e-05 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 39/33 ECSubWrite(tid=81, reqid=client.4646.0:292, at_version=39'42, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:24.891975+0000", 2024-09-16T04:01:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.236702491, 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022900910000000002, 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.891975+0000", 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.891975+0000", 2024-09-16T04:01:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.891976+0000", 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.747e-06 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.892205+0000", 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000228099 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:26.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.892237+0000", 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2805000000000002e-05 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.892254+0000", 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6172999999999999e-05 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.892424+0000", 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017050400000000001 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.892438+0000", 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4185999999999999e-05 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894178+0000", 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001740166 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894265+0000", 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6410999999999996e-05 2024-09-16T04:01:26.155 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 39/33 ECSubWrite(tid=82, reqid=client.4646.0:293, at_version=39'43, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:24.894169+0000", 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.2345077469999999, 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001450766, 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894169+0000", 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894169+0000", 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894172+0000", 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6800000000000002e-06 2024-09-16T04:01:26.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894217+0000", 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4755000000000001e-05 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894264+0000", 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6955e-05 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894280+0000", 2024-09-16T04:01:26.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6019999999999999e-05 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894387+0000", 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106983 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.894400+0000", 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3579e-05 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:26.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.895505+0000", 2024-09-16T04:01:26.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001104502 2024-09-16T04:01:26.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.895620+0000", 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011529199999999999 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 39/33 ECSubWrite(tid=83, reqid=client.4646.0:293, at_version=0'0, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:24.898313+0000", 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.23036448, 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013966510000000001, 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:26.161 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.898313+0000", 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.898313+0000", 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:26.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.898314+0000", 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.646e-06 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.898335+0000", 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0582999999999999e-05 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.898366+0000", 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1355e-05 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:26.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.898382+0000", 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5668999999999999e-05 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.898522+0000", 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014000199999999999 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:26.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.898536+0000", 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4113e-05 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.899624+0000", 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010877090000000001 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.899709+0000", 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5574000000000004e-05 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.165 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.166 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.166 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.166 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.166 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:26.167 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:01:26.168 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:26.166+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:26.269 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:26.266+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:26.295 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:26.295 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:26.295 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:26.295 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:26.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:01:26.370 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:26.369+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:26.431 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:26.431 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:26.431 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:26.431 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:26.431 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:26.444 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:01:26.489 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:26.486+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:26.573 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:26.573 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:26.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:26.573 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:26.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.573 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 40/33 ECSubWrite(tid=66, reqid=client.4646.0:302, at_version=40'8, trim_to=0'0, roll_forward_to=37'6))", 2024-09-16T04:01:26.573 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:25.919876+0000", 2024-09-16T04:01:26.573 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.65223205799999995, 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025883859999999998, 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.919876+0000", 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:26.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.919876+0000", 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.919877+0000", 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920263+0000", 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00038648099999999999 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920308+0000", 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4641999999999997e-05 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920338+0000", 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0185999999999999e-05 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920466+0000", 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012813000000000001 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920483+0000", 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7255e-05 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.922353+0000", 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001870086 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.922464+0000", 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000110943 2024-09-16T04:01:26.577 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 40/33 ECSubWrite(tid=67, reqid=client.4646.0:302, at_version=0'0, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:25.924455+0000", 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.64765345200000002, 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082016599999999999, 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:26.578 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.924455+0000", 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.924455+0000", 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.924456+0000", 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1829999999999999e-06 2024-09-16T04:01:26.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.924468+0000", 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2394e-05 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.924485+0000", 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6755000000000001e-05 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:26.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.924495+0000", 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8800000000000003e-06 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.924581+0000", 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6230999999999995e-05 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.924598+0000", 2024-09-16T04:01:26.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7238e-05 2024-09-16T04:01:26.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:26.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.925225+0000", 2024-09-16T04:01:26.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062682999999999999 2024-09-16T04:01:26.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:26.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:26.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:26.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.925275+0000", 2024-09-16T04:01:26.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9654999999999998e-05 2024-09-16T04:01:26.585 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.585 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.585 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.585 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:26.585 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:26.585 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:26.585 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:26.565+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:26.587 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:01:26.643 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:26.673 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:26.666+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:26.750 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:26.750 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:26.750 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:26.750 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:26.765 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:01:26.803 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:26.803 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:26.809 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:26.806+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:26.822 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:26.822 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:26.843 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:26.843 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:26.864 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:26.864 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:26.875 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:26.870+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:26.882 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:26.882 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:26.883 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:127: done (5 left) 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:128: rollback oid 40 current snap is 3 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 40 to 1 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:122: finishing write tid 1 to smithi18831615-3 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:122: finishing write tid 2 to smithi18831615-3 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:122: finishing write tid 3 to smithi18831615-3 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:122: finishing write tid 4 to smithi18831615-3 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:122: finishing write tid 5 to smithi18831615-3 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:122: oid 3 updating version 0 to 24 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:122: oid 3 updating version 24 to 26 2024-09-16T04:01:26.890 INFO:tasks.rados.rados.0.smithi188.stdout:122: oid 3 version 26 is already newer than 25 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:122: oid 3 version 26 is already newer than 23 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 3 v 26 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:122: left oid 3 (ObjNum 73 snap 3 seq_num 73) 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: finishing write tid 1 to smithi18831615-50 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: finishing write tid 2 to smithi18831615-50 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: finishing write tid 3 to smithi18831615-50 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: finishing write tid 4 to smithi18831615-50 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: finishing write tid 5 to smithi18831615-50 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: oid 50 updating version 0 to 16 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: oid 50 updating version 16 to 17 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: oid 50 updating version 17 to 18 2024-09-16T04:01:26.891 INFO:tasks.rados.rados.0.smithi188.stdout:123: oid 50 updating version 18 to 19 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 50 v 19 (ObjNum 74 snap 3 seq_num 74) dirty exists 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:123: left oid 50 (ObjNum 74 snap 3 seq_num 74) 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:124: finishing write tid 1 to smithi18831615-36 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:124: finishing write tid 2 to smithi18831615-36 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:122: done (5 left) 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:123: done (4 left) 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:129: delete oid 3 current snap is 3 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:124: finishing write tid 3 to smithi18831615-36 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:124: finishing write tid 4 to smithi18831615-36 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:124: oid 36 updating version 0 to 15 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:124: oid 36 version 15 is already newer than 14 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:124: oid 36 version 15 is already newer than 13 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 36 v 15 (ObjNum 75 snap 3 seq_num 75) dirty exists 2024-09-16T04:01:26.892 INFO:tasks.rados.rados.0.smithi188.stdout:124: left oid 36 (ObjNum 75 snap 3 seq_num 75) 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: finishing write tid 1 to smithi18831615-17 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: finishing write tid 2 to smithi18831615-17 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: finishing write tid 3 to smithi18831615-17 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: finishing write tid 4 to smithi18831615-17 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: finishing write tid 5 to smithi18831615-17 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: oid 17 updating version 0 to 13 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: oid 17 updating version 13 to 15 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: oid 17 updating version 15 to 16 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: oid 17 version 16 is already newer than 14 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 17 v 16 (ObjNum 76 snap 3 seq_num 76) dirty exists 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:125: left oid 17 (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:126: finishing rollback tid 0 to smithi18831615-49 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:126: finishing rollback tid 1 to smithi18831615-49 2024-09-16T04:01:26.893 INFO:tasks.rados.rados.0.smithi188.stdout:126: finishing rollback tid 2 to smithi18831615-49 2024-09-16T04:01:26.894 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 49 v 30 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T04:01:26.900 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:26.900 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:26.901 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:26.901 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:26.901 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:26.909 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T04:01:26.917 INFO:tasks.rados.rados.0.smithi188.stdout:128: finishing rollback tid 0 to smithi18831615-40 2024-09-16T04:01:26.918 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:26.919 INFO:tasks.rados.rados.0.smithi188.stdout:128: finishing rollback tid 1 to smithi18831615-40 2024-09-16T04:01:26.922 INFO:tasks.rados.rados.0.smithi188.stdout:128: finishing rollback tid 2 to smithi18831615-40 2024-09-16T04:01:26.923 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 40 v 34 (ObjNum 33 snap 0 seq_num 3898288432) dirty exists 2024-09-16T04:01:26.923 INFO:tasks.rados.rados.0.smithi188.stdout:124: done (4 left) 2024-09-16T04:01:26.923 INFO:tasks.rados.rados.0.smithi188.stdout:125: done (3 left) 2024-09-16T04:01:26.923 INFO:tasks.rados.rados.0.smithi188.stdout:126: done (2 left) 2024-09-16T04:01:26.923 INFO:tasks.rados.rados.0.smithi188.stdout:128: done (1 left) 2024-09-16T04:01:26.923 INFO:tasks.rados.rados.0.smithi188.stdout:129: done (0 left) 2024-09-16T04:01:26.923 INFO:tasks.rados.rados.0.smithi188.stdout:130: copy_from oid 43 from oid 46 current snap is 3 2024-09-16T04:01:26.924 INFO:tasks.rados.rados.0.smithi188.stdout:131: snap_remove snap 1 2024-09-16T04:01:26.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:01:26.949 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:26.949 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:26.975 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:26.969+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:26.979 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:26.979 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:26.996 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:26.996 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:27.014 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:27.014 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:27.035 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:27.035 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:27.059 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:27.059 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:27.075 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:27.077 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:27.074+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:27.095 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:27.096 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:27.096 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:27.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:01:27.116 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:27.116 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:27.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:27.113+0000 7fac850bc640 0 -- 172.21.15.37:0/2405197020 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7fac802268d0 msgr2=0x7fac80228cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:27.137 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:27.137 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:27.176 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:27.174+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:27.241 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:27.242 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:27.256 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:01:27.277 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:27.274+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:27.379 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:27.378+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:27.420 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:27.420 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:27.434 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:01:27.479 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:27.474+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:27.552 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:27.552 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:27.566 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:01:27.580 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:27.578+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:27.680 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:27.674+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:27.682 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:27.682 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:27.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:01:27.781 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:27.773+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:27.815 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:27.815 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:27.829 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:01:27.882 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:27.877+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:27.892 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T04:01:27.898 INFO:tasks.rados.rados.0.smithi188.stdout:131: done (1 left) 2024-09-16T04:01:27.898 INFO:tasks.rados.rados.0.smithi188.stdout:132: rmattr oid 23 current snap is 3 2024-09-16T04:01:27.898 INFO:tasks.rados.rados.0.smithi188.stdout:132: done (1 left) 2024-09-16T04:01:27.898 INFO:tasks.rados.rados.0.smithi188.stdout:133: snap_create 2024-09-16T04:01:27.899 INFO:tasks.rados.rados.0.smithi188.stdout:130: finishing copy_from racing read to smithi18831615-43 2024-09-16T04:01:27.899 INFO:tasks.rados.rados.0.smithi188.stdout:130: finishing copy_from to smithi18831615-43 2024-09-16T04:01:27.899 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 43 v 17 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-09-16T04:01:27.914 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:01:27.927 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T04:01:27.958 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:27.959 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:27.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:01:28.021 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:28.017+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:28.083 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:28.078+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:28.092 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:28.092 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:28.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:01:28.185 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:28.178+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:28.228 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:28.228 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:28.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:01:28.285 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:28.282+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:28.365 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:28.365 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:28.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:01:28.387 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:28.385+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:28.488 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:28.482+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:28.502 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:28.502 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:28.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:01:28.608 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:28.606+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:28.684 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:28.684 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:28.690 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:28.685+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:28.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:01:28.791 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:28.790+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:28.865 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:28.865 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:28.879 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:01:28.891 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:28.889+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:28.896 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-09-16T04:01:28.896 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T04:01:28.902 INFO:tasks.rados.rados.0.smithi188.stdout:130: done (1 left) 2024-09-16T04:01:28.902 INFO:tasks.rados.rados.0.smithi188.stdout:133: done (0 left) 2024-09-16T04:01:28.902 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 39 current snap is 4 2024-09-16T04:01:28.902 INFO:tasks.rados.rados.0.smithi188.stdout:134: seq_num 78 ranges {0=557056,557056=671744,1228800=98304} 2024-09-16T04:01:28.910 INFO:tasks.rados.rados.0.smithi188.stdout:134: writing smithi18831615-39 from 0 to 557056 tid 1 2024-09-16T04:01:28.920 INFO:tasks.rados.rados.0.smithi188.stdout:134: writing smithi18831615-39 from 557056 to 1228800 tid 2 2024-09-16T04:01:28.922 INFO:tasks.rados.rados.0.smithi188.stdout:134: writing smithi18831615-39 from 1228800 to 1327104 tid 3 2024-09-16T04:01:28.922 INFO:tasks.rados.rados.0.smithi188.stdout:135: read oid 50 snap -1 2024-09-16T04:01:28.922 INFO:tasks.rados.rados.0.smithi188.stdout:135: expect (ObjNum 74 snap 3 seq_num 74) 2024-09-16T04:01:28.923 INFO:tasks.rados.rados.0.smithi188.stdout:136: copy_from oid 25 from oid 41 current snap is 4 2024-09-16T04:01:28.923 INFO:tasks.rados.rados.0.smithi188.stdout:134: finishing write tid 1 to smithi18831615-39 2024-09-16T04:01:28.923 INFO:tasks.rados.rados.0.smithi188.stdout:137: snap_create 2024-09-16T04:01:28.928 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T04:01:28.929 INFO:tasks.rados.rados.0.smithi188.stdout:134: finishing write tid 2 to smithi18831615-39 2024-09-16T04:01:28.933 INFO:tasks.rados.rados.0.smithi188.stdout:136: finishing copy_from racing read to smithi18831615-25 2024-09-16T04:01:28.933 INFO:tasks.rados.rados.0.smithi188.stdout:134: finishing write tid 3 to smithi18831615-39 2024-09-16T04:01:28.933 INFO:tasks.rados.rados.0.smithi188.stdout:134: finishing write tid 4 to smithi18831615-39 2024-09-16T04:01:28.936 INFO:tasks.rados.rados.0.smithi188.stdout:134: finishing write tid 5 to smithi18831615-39 2024-09-16T04:01:28.936 INFO:tasks.rados.rados.0.smithi188.stdout:134: oid 39 updating version 0 to 21 2024-09-16T04:01:28.936 INFO:tasks.rados.rados.0.smithi188.stdout:134: oid 39 updating version 21 to 23 2024-09-16T04:01:28.936 INFO:tasks.rados.rados.0.smithi188.stdout:134: oid 39 updating version 23 to 24 2024-09-16T04:01:28.936 INFO:tasks.rados.rados.0.smithi188.stdout:134: oid 39 version 24 is already newer than 22 2024-09-16T04:01:28.937 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 39 v 24 (ObjNum 78 snap 4 seq_num 78) dirty exists 2024-09-16T04:01:28.937 INFO:tasks.rados.rados.0.smithi188.stdout:134: left oid 39 (ObjNum 78 snap 4 seq_num 78) 2024-09-16T04:01:28.937 INFO:tasks.rados.rados.0.smithi188.stdout:136: finishing copy_from to smithi18831615-25 2024-09-16T04:01:28.937 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 25 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T04:01:28.940 INFO:tasks.rados.rados.0.smithi188.stdout:135: expect (ObjNum 74 snap 3 seq_num 74) 2024-09-16T04:01:28.992 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:28.990+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:29.016 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:29.016 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:29.030 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:01:29.093 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:29.085+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:29.156 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:01:29.194 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:29.190+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:29.204 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:29.204 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:29.219 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:01:29.294 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:29.290+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:29.305 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:29.305 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:29.328 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:29.329 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:29.336 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:29.336 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:29.350 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:01:29.351 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:29.351 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:29.369 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:29.369 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:29.387 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:29.387 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:29.395 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:29.390+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:29.409 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:29.429 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:29.429 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:29.451 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:29.451 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:29.473 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:29.473 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:29.495 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:29.496 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:29.496 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:29.490+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:29.518 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:29.518 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:29.525 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:29.525 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:29.525 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:01:29.525 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:29.539 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:01:29.540 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:29.540 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:29.562 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:29.562 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:29.586 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:29.586 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:29.596 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:29.589+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:29.610 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:29.610 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:29.632 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:29.632 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:29.698 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:29.694+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:29.711 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:29.711 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:29.711 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:01:29.711 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:01:29.711 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:29.726 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:01:29.818 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:29.814+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:29.895 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 40/33 ECSubWrite(tid=18, reqid=client.4646.0:314, at_version=40'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.014390+0000", 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.8783013319999999, 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026800579999999999, 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.896 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014390+0000", 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014390+0000", 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014390+0000", 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3000000000000001e-07 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014633+0000", 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000242308 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014662+0000", 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9785999999999999e-05 2024-09-16T04:01:29.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014679+0000", 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6898999999999999e-05 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014778+0000", 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9059000000000001e-05 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014791+0000", 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2655e-05 2024-09-16T04:01:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017027+0000", 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022360739999999998 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017070+0000", 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2747000000000003e-05 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 40/33 ECSubWrite(tid=19, reqid=client.4646.0:315, at_version=40'14, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.020273+0000", 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.8724179369999998, 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022028460000000001, 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020273+0000", 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020273+0000", 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020275+0000", 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.08e-06 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020620+0000", 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034477599999999998 2024-09-16T04:01:29.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020654+0000", 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3726000000000002e-05 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020678+0000", 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4243000000000002e-05 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020811+0000", 2024-09-16T04:01:29.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013336700000000001 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020830+0000", 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8606000000000001e-05 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022434+0000", 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00160406 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022476+0000", 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1987999999999998e-05 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 40/33 ECSubWrite(tid=20, reqid=client.4646.0:316, at_version=40'15, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.023237+0000", 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.8694538810000001, 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001753957, 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.905 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023237+0000", 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023237+0000", 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023239+0000", 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7400000000000001e-06 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023283+0000", 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4091999999999997e-05 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023320+0000", 2024-09-16T04:01:29.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6681999999999999e-05 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023344+0000", 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4049999999999998e-05 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023523+0000", 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017874300000000001 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023551+0000", 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8782e-05 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024931+0000", 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013794980000000001 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024991+0000", 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.037e-05 2024-09-16T04:01:29.909 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 40/33 ECSubWrite(tid=21, reqid=client.4646.0:316, at_version=0'0, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.028518+0000", 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.864172886, 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001698976, 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028518+0000", 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028518+0000", 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028520+0000", 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7230000000000001e-06 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028565+0000", 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5281999999999998e-05 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028593+0000", 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.775e-05 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028677+0000", 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4327999999999996e-05 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028833+0000", 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015589999999999999 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028859+0000", 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5613e-05 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.030128+0000", 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012691390000000001 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.030217+0000", 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9240999999999995e-05 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 41/33 ECSubWrite(tid=69, reqid=osd.4.0:69, at_version=41'20, trim_to=0'0, roll_forward_to=37'18))", 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.899386+0000", 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9933048429999998, 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003829781, 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.914 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.899386+0000", 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.899386+0000", 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.899387+0000", 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7e-07 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.899446+0000", 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9382999999999997e-05 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.899479+0000", 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2981999999999997e-05 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.899502+0000", 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2713000000000001e-05 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.899656+0000", 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000154043 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.899671+0000", 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5662999999999999e-05 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.903125+0000", 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034531050000000002 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.903216+0000", 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1521999999999996e-05 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=23, reqid=osd.15.0:23, at_version=41'17, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.904709+0000", 2024-09-16T04:01:29.918 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.987982015, 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054382349999999996, 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904709+0000", 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904709+0000", 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904710+0000", 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3e-07 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904764+0000", 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3995000000000001e-05 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904795+0000", 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1593000000000001e-05 2024-09-16T04:01:29.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904818+0000", 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2133e-05 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.905345+0000", 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000527171 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.905370+0000", 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5633999999999999e-05 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910074+0000", 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00470339 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910147+0000", 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3589000000000003e-05 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.922 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=41'19, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.907264+0000", 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9854271940000001, 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060462780000000004, 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907264+0000", 2024-09-16T04:01:29.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907264+0000", 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907266+0000", 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8530000000000001e-06 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907317+0000", 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0806000000000001e-05 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907355+0000", 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8262000000000001e-05 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907380+0000", 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4776000000000001e-05 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907566+0000", 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018675300000000001 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907592+0000", 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5775999999999999e-05 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.913228+0000", 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0056355909999999997 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.913310+0000", 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2460999999999995e-05 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 41/33 ECSubWrite(tid=71, reqid=osd.4.0:69, at_version=0'0, trim_to=0'0, roll_forward_to=41'20))", 2024-09-16T04:01:29.927 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.907893+0000", 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.984798541, 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054840339999999996, 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907893+0000", 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907893+0000", 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907894+0000", 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.883e-06 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907922+0000", 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7702999999999999e-05 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907958+0000", 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5339000000000002e-05 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907980+0000", 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.234e-05 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.908186+0000", 2024-09-16T04:01:29.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020639500000000001 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.908211+0000", 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4666000000000001e-05 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.913295+0000", 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050841799999999998 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.913377+0000", 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1527999999999996e-05 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 41/33 ECSubWriteReply(tid=28, last_complete=41'17, committed=1, applied=1))", 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.911122+0000", 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.981568701, 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00064027599999999995, 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.932 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911122+0000", 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911122+0000", 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911124+0000", 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1799999999999999e-06 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911132+0000", 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6859999999999995e-06 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911156+0000", 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3286e-05 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911173+0000", 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7516999999999998e-05 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911713+0000", 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053992500000000002 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911763+0000", 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9682000000000003e-05 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.935 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 41/33 ECSubWriteReply(tid=28, last_complete=41'17, committed=1, applied=1))", 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.912653+0000", 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9800383880000001, 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030283000000000003, 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.912653+0000", 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.912653+0000", 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.912655+0000", 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9300000000000002e-06 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.912666+0000", 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1653e-05 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.912722+0000", 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5838999999999998e-05 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.912742+0000", 2024-09-16T04:01:29.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9649000000000001e-05 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.912910+0000", 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000168197 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.912956+0000", 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5562000000000002e-05 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 41/33 ECSubWriteReply(tid=28, last_complete=41'17, committed=1, applied=1))", 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.915138+0000", 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.977553602, 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00071025799999999996, 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915138+0000", 2024-09-16T04:01:29.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915138+0000", 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915140+0000", 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.12e-06 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915151+0000", 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1456e-05 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915179+0000", 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7872999999999999e-05 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915200+0000", 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0582999999999999e-05 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915548+0000", 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034803600000000003 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915848+0000", 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030018999999999998 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 41/33 ECSubWriteReply(tid=29, last_complete=41'19, committed=1, applied=1))", 2024-09-16T04:01:29.943 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.915317+0000", 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9773739620000002, 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00064187300000000005, 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915317+0000", 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915317+0000", 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915318+0000", 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3e-06 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915327+0000", 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1729999999999995e-06 2024-09-16T04:01:29.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915349+0000", 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2673000000000001e-05 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915381+0000", 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1272000000000001e-05 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915914+0000", 2024-09-16T04:01:29.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053374200000000003 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915959+0000", 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4712999999999997e-05 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 41/33 ECSubWriteReply(tid=29, last_complete=41'19, committed=1, applied=1))", 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.915596+0000", 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9770947679999997, 2024-09-16T04:01:29.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046240300000000001, 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915596+0000", 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915596+0000", 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915598+0000", 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.86e-06 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915610+0000", 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2109e-05 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915640+0000", 2024-09-16T04:01:29.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9879999999999999e-05 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.915660+0000", 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9939e-05 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916017+0000", 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00035713900000000002 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916059+0000", 2024-09-16T04:01:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1476000000000001e-05 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.as1 41/33 ECSubWrite(tid=13, reqid=osd.3.0:13, at_version=41'8, trim_to=0'0, roll_forward_to=38'6))", 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.917371+0000", 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9753196499999999, 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032756550000000001, 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917371+0000", 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917371+0000", 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917373+0000", 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.886e-06 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917446+0000", 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2649000000000002e-05 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917494+0000", 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7988999999999997e-05 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917517+0000", 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2932999999999999e-05 2024-09-16T04:01:29.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917747+0000", 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023036500000000001 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917772+0000", 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4473000000000001e-05 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920552+0000", 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027804919999999999 2024-09-16T04:01:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920647+0000", 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4868e-05 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 41/33 ECSubWriteReply(tid=29, last_complete=41'19, committed=1, applied=1))", 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.917888+0000", 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9748035850000001, 2024-09-16T04:01:29.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001896289, 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917888+0000", 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917888+0000", 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917896+0000", 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7499999999999992e-06 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917909+0000", 2024-09-16T04:01:29.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3094e-05 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917936+0000", 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6421999999999999e-05 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917954+0000", 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8227e-05 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918166+0000", 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021229800000000001 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.919784+0000", 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001617498 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s1 41/33 ECSubWrite(tid=198, reqid=osd.13.0:198, at_version=41'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.920987+0000", 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9717040849999998, 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055428689999999997, 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920987+0000", 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920987+0000", 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920988+0000", 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.37e-07 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921042+0000", 2024-09-16T04:01:29.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4561999999999997e-05 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921116+0000", 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3955000000000006e-05 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921139+0000", 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3108999999999998e-05 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921291+0000", 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015138 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921315+0000", 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3747000000000002e-05 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.926443+0000", 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051287240000000003 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.926530+0000", 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6655000000000003e-05 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=25, reqid=osd.15.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=41'19))", 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.922788+0000", 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9699027469999999, 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003668035, 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.964 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922788+0000", 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922788+0000", 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922791+0000", 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.199e-06 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922820+0000", 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9099999999999999e-05 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922854+0000", 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4269e-05 2024-09-16T04:01:29.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922877+0000", 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2863000000000001e-05 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.923084+0000", 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020720499999999999 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.923108+0000", 2024-09-16T04:01:29.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4136999999999999e-05 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.926408+0000", 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00330029 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.926456+0000", 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7972000000000001e-05 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.968 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 41/33 ECSubWriteReply(tid=30, last_complete=41'19, committed=1, applied=1))", 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.925749+0000", 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9669422189999999, 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032755000000000002, 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.925749+0000", 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.925749+0000", 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.925751+0000", 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2400000000000002e-06 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.925763+0000", 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2300000000000001e-05 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.925794+0000", 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0516000000000001e-05 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.925814+0000", 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9633e-05 2024-09-16T04:01:29.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.926026+0000", 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021218499999999999 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.926076+0000", 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0676000000000001e-05 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s1 41/33 ECSubWrite(tid=200, reqid=osd.13.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.929140+0000", 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.963551029, 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039897700000000001, 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929140+0000", 2024-09-16T04:01:29.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929140+0000", 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929142+0000", 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9360000000000002e-06 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929171+0000", 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.898e-05 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929211+0000", 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9779e-05 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929237+0000", 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6529e-05 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929384+0000", 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014694999999999999 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929408+0000", 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3604e-05 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.933037+0000", 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036292709999999999 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.933130+0000", 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2720999999999995e-05 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.as1 41/33 ECSubWrite(tid=14, reqid=osd.3.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=41'8))", 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.931674+0000", 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9610175490000001, 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047356760000000003, 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:29.977 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931674+0000", 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931674+0000", 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:29.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931675+0000", 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.223e-06 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931690+0000", 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5158e-05 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:29.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931712+0000", 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1861e-05 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931724+0000", 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2609000000000001e-05 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931799+0000", 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4882000000000006e-05 2024-09-16T04:01:29.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931811+0000", 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2164000000000001e-05 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.936322+0000", 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045109750000000004 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:29.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.936409+0000", 2024-09-16T04:01:29.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6804000000000004e-05 2024-09-16T04:01:29.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.982 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:29.982 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:29.982 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T04:01:29.984 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:01:29.985 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:134: done (3 left) 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:135: done (2 left) 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:136: done (1 left) 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:137: done (0 left) 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:138: read oid 17 snap -1 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:138: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:139: read oid 42 snap -1 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:139: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:29.986 INFO:tasks.rados.rados.0.smithi188.stdout:140: delete oid 18 current snap is 5 2024-09-16T04:01:29.987 INFO:tasks.rados.rados.0.smithi188.stdout:139: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:29.987 INFO:tasks.rados.rados.0.smithi188.stdout:138: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:01:29.987 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:29.978+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:30.035 INFO:tasks.rados.rados.0.smithi188.stdout:138: done (2 left) 2024-09-16T04:01:30.035 INFO:tasks.rados.rados.0.smithi188.stdout:139: done (1 left) 2024-09-16T04:01:30.035 INFO:tasks.rados.rados.0.smithi188.stdout:140: done (0 left) 2024-09-16T04:01:30.035 INFO:tasks.rados.rados.0.smithi188.stdout:141: snap_create 2024-09-16T04:01:30.085 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:30.082+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:30.144 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:30.144 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:30.144 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:01:30.144 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:30.158 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:01:30.186 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:30.182+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:30.287 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:30.286+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:30.329 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:30.329 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:30.329 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:01:30.329 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:01:30.329 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:30.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:30.325+0000 7f19ad872640 0 --2- 172.21.15.37:0/2892134527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1994003a40 0x7f19a81f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:01:30.343 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:01:30.390 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:30.386+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:30.465 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:30.465 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:01:30.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:01:30.465 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:01:30.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.465 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:291 3.cs0 3:3631f100:::smithi18831615-24:head [append 0~802816] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T04:01:30.465 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:24.884471+0000", 2024-09-16T04:01:30.465 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.5787171569999998, 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011834086000000001, 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 291 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.466 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.884471+0000", 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.884471+0000", 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.884474+0000", 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.067e-06 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.885509+0000", 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010356860000000001 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.885544+0000", 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4876000000000003e-05 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.885561+0000", 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7184999999999999e-05 2024-09-16T04:01:30.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.885715+0000", 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000153382 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.885933+0000", 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000218233 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.889127+0000", 2024-09-16T04:01:30.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031940089999999998 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.895318+0000", 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061913719999999997 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.896110+0000", 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079168800000000003 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.896305+0000", 2024-09-16T04:01:30.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019458799999999999 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:294 3.cs0 3:3631f100:::smithi18831615-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:24.890467+0000", 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.5727210149999999, 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011314296999999999, 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.471 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 294 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.890467+0000", 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.890467+0000", 2024-09-16T04:01:30.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.890468+0000", 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0999999999999997e-07 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.890471+0000", 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4230000000000001e-06 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.890482+0000", 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0933e-05 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.890492+0000", 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9029999999999999e-06 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.896735+0000", 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0062427580000000002 2024-09-16T04:01:30.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.896878+0000", 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014392699999999999 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.899262+0000", 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023831030000000001 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.899402+0000", 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014022399999999999 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:24.901781+0000", 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002379216 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:297 3.cs0 3:3922c872:::smithi18831615-21:head [append 0~425984] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.476 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.899819+0000", 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5633691980000002, 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020147297000000002, 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 297 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.899819+0000", 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.899819+0000", 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.899822+0000", 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0029999999999999e-06 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900319+0000", 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049783800000000001 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900351+0000", 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1365999999999998e-05 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900368+0000", 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6840000000000001e-05 2024-09-16T04:01:30.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900494+0000", 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012627999999999999 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900509+0000", 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5342999999999999e-05 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.904460+0000", 2024-09-16T04:01:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039507700000000001 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.904668+0000", 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000208285 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.906930+0000", 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002261692 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.919087+0000", 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012156752999999999 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.919839+0000", 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075222699999999995 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.919966+0000", 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012689999999999999 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.482 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:298 3.cs0 3:3922c872:::smithi18831615-21:head [append 0~573440] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.908721+0000", 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5544671609999998, 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01288924, 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.483 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 298 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.908721+0000", 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.908721+0000", 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.908723+0000", 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.216e-06 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909354+0000", 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063166899999999996 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909391+0000", 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6763e-05 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909420+0000", 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9179e-05 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.915299+0000", 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058785859999999999 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.915565+0000", 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026633700000000001 2024-09-16T04:01:30.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.917241+0000", 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016752119999999999 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.920302+0000", 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030614969999999998 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.921510+0000", 2024-09-16T04:01:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012080229999999999 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.921610+0000", 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9758000000000003e-05 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 40/33 ECSubWriteReply(tid=85, last_complete=40'45, committed=1, applied=1))", 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.909842+0000", 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5533458570000001, 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.008991878, 2024-09-16T04:01:30.488 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909842+0000", 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909842+0000", 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909844+0000", 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1260000000000001e-06 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909855+0000", 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1027e-05 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909917+0000", 2024-09-16T04:01:30.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1785000000000006e-05 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.909935+0000", 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8335999999999999e-05 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.918757+0000", 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0088219490000000008 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.918834+0000", 2024-09-16T04:01:30.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6655000000000004e-05 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:299 3.cs0 3:3922c872:::smithi18831615-21:head [append 0~196608] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.911542+0000", 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5516461259999996, 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016481741000000001, 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.492 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 299 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911542+0000", 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911542+0000", 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911544+0000", 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3800000000000001e-06 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911849+0000", 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030538000000000001 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911878+0000", 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911891+0000", 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3200000000000001e-05 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.922448+0000", 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010556777 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.922573+0000", 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012551399999999999 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.923603+0000", 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001029234 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.926788+0000", 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003185799 2024-09-16T04:01:30.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.927884+0000", 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001095295 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928023+0000", 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013962299999999999 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 40/33 ECSubWriteReply(tid=85, last_complete=40'45, committed=1, applied=1))", 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.911837+0000", 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5513507820000001, 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081520840000000004, 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911837+0000", 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911837+0000", 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911846+0000", 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.799e-06 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911855+0000", 2024-09-16T04:01:30.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0929999999999992e-06 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911880+0000", 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.544e-05 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911897+0000", 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6396e-05 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.919604+0000", 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007707205 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.919989+0000", 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038515100000000003 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:300 3.cs0 3:3922c872:::smithi18831615-21:head [setxattr _header (56) in=7b] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.501 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.911913+0000", 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5512744139999999, 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019281432000000001, 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 300 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911913+0000", 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911913+0000", 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911914+0000", 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4600000000000003e-07 2024-09-16T04:01:30.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911920+0000", 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4500000000000003e-06 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911935+0000", 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5330000000000001e-05 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911945+0000", 2024-09-16T04:01:30.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8530000000000004e-06 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.924549+0000", 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012604087 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.924819+0000", 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000269597 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.926100+0000", 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012815540000000001 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928427+0000", 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023271569999999998 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931023+0000", 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025953780000000002 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931195+0000", 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017207999999999999 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:301 3.cs0 3:3922c872:::smithi18831615-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5512304690000001, 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024704912999999998, 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 301 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911958+0000", 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911962+0000", 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.63e-06 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911976+0000", 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4027e-05 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911985+0000", 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7530000000000007e-06 2024-09-16T04:01:30.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928563+0000", 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016578809 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928680+0000", 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117123 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931348+0000", 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002667141 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931647+0000", 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029889700000000001 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.936662+0000", 2024-09-16T04:01:30.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050156630000000001 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:303 3.cs0 3:385a92e2:::smithi18831615-4:head [delete] snapc 4=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.918817+0000", 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5443709370000001, 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014517044999999999, 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 303 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.918817+0000", 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.918817+0000", 2024-09-16T04:01:30.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.918819+0000", 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5720000000000001e-06 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.918829+0000", 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3139999999999993e-06 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.918876+0000", 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6935999999999999e-05 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.918890+0000", 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4851000000000001e-05 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928755+0000", 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098647500000000003 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928904+0000", 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000149254 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.929914+0000", 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010099029999999999 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.932499+0000", 2024-09-16T04:01:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025841620000000001 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.933188+0000", 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068922700000000005 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.933334+0000", 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000146076 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:309 3.fs0 3:f669afff:::smithi18831615-50:head [append 0~475136] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.988508+0000", 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.4746795849999996, 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0096177629999999997, 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 309 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.519 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.988508+0000", 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.988508+0000", 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.988512+0000", 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1069999999999998e-06 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.988888+0000", 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037549099999999999 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.988924+0000", 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6739000000000002e-05 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.988942+0000", 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7274000000000001e-05 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.989070+0000", 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000128493 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.989234+0000", 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016422299999999999 2024-09-16T04:01:30.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.991142+0000", 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001908 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.993726+0000", 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025833010000000001 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.997997+0000", 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042711199999999998 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.998126+0000", 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012901500000000001 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:310 3.fs0 3:f669afff:::smithi18831615-50:head [append 0~622592] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.524 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.994035+0000", 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.4691527300000002, 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0088128129999999992, 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 310 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.994035+0000", 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.994035+0000", 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.994038+0000", 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5160000000000001e-06 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.994870+0000", 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00083248800000000004 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.994898+0000", 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8323e-05 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.994912+0000", 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.38e-05 2024-09-16T04:01:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995027+0000", 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114617 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995183+0000", 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000156582 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.996545+0000", 2024-09-16T04:01:30.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013619859999999999 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.002304+0000", 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057583030000000002 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.002731+0000", 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000427081 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.002848+0000", 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117117 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:311 3.fs0 3:f669afff:::smithi18831615-50:head [append 0~147456] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.995314+0000", 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.4678734030000005, 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0084987159999999999, 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.530 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 311 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995314+0000", 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995314+0000", 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995316+0000", 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.007e-06 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995432+0000", 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000116094 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995454+0000", 2024-09-16T04:01:30.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1146e-05 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995469+0000", 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5339999999999999e-05 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.998267+0000", 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027978429999999999 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.998558+0000", 2024-09-16T04:01:30.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029116299999999999 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.000089+0000", 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001531103 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.002305+0000", 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022161440000000002 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.003736+0000", 2024-09-16T04:01:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001431009 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.003813+0000", 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6866999999999994e-05 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:312 3.fs0 3:f669afff:::smithi18831615-50:head [setxattr _header (56) in=7b] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.995509+0000", 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.467678297, 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011078628, 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.536 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 312 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995509+0000", 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995509+0000", 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995511+0000", 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4100000000000001e-06 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995517+0000", 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.55e-06 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995533+0000", 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5688999999999998e-05 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995548+0000", 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4423999999999999e-05 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.001129+0000", 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055812420000000001 2024-09-16T04:01:30.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.001233+0000", 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103878 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.001769+0000", 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053586900000000002 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.003572+0000", 2024-09-16T04:01:30.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018036459999999999 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.006496+0000", 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029235749999999999 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.006588+0000", 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2344999999999998e-05 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:313 3.fs0 3:f669afff:::smithi18831615-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.995566+0000", 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.467621415, 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013917357, 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 313 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.542 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995566+0000", 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995566+0000", 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995567+0000", 2024-09-16T04:01:30.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995572+0000", 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9599999999999999e-06 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995584+0000", 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1593000000000001e-05 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.995592+0000", 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5599999999999994e-06 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.003450+0000", 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078577370000000001 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.003565+0000", 2024-09-16T04:01:30.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114474 2024-09-16T04:01:30.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.006999+0000", 2024-09-16T04:01:30.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034341039999999999 2024-09-16T04:01:30.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.007167+0000", 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016799999999999999 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.009484+0000", 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002317066 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s6 40/33 ECSubWrite(tid=191, reqid=client.4646.0:318, at_version=40'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.028979+0000", 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.4342085510000002, 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0095819080000000001, 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028979+0000", 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028979+0000", 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028981+0000", 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8920000000000001e-06 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.030358+0000", 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766329999999999 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.030388+0000", 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0704000000000003e-05 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.030401+0000", 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2136e-05 2024-09-16T04:01:30.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.030482+0000", 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1204000000000004e-05 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.030491+0000", 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6229999999999992e-06 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.038522+0000", 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080305470000000007 2024-09-16T04:01:30.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.038561+0000", 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9168999999999997e-05 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.1s1 41/33 ECSubWrite(tid=72, reqid=osd.4.0:70, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.916064+0000", 2024-09-16T04:01:30.554 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.5471233770000001, 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012460000000000001, 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916064+0000", 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916064+0000", 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916066+0000", 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.95e-06 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916095+0000", 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.847e-05 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916126+0000", 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1232000000000002e-05 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916149+0000", 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3232999999999999e-05 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916512+0000", 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000362612 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.916526+0000", 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4497e-05 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.928435+0000", 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011908363999999999 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.928524+0000", 2024-09-16T04:01:30.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9642000000000007e-05 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.as6 41/33 ECSubWrite(tid=13, reqid=osd.3.0:13, at_version=41'8, trim_to=0'0, roll_forward_to=38'6))", 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.917891+0000", 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.5452971629999999, 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012130472999999999, 2024-09-16T04:01:30.559 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917891+0000", 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917891+0000", 2024-09-16T04:01:30.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917891+0000", 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3300000000000001e-07 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.917962+0000", 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0698999999999995e-05 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918002+0000", 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9523000000000001e-05 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918027+0000", 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5012e-05 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918250+0000", 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000223122 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918280+0000", 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0202999999999999e-05 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.929940+0000", 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011659683000000001 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930021+0000", 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1397999999999996e-05 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.563 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s6 41/33 ECSubWrite(tid=197, reqid=osd.13.0:197, at_version=41'18, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.919994+0000", 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.5431932550000003, 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010144363, 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.564 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.919994+0000", 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.919994+0000", 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.919996+0000", 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9300000000000002e-06 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920046+0000", 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9765000000000002e-05 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920077+0000", 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1109999999999999e-05 2024-09-16T04:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920099+0000", 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1418999999999999e-05 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920289+0000", 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018986599999999999 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920315+0000", 2024-09-16T04:01:30.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6242999999999999e-05 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930030+0000", 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097152079999999995 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930139+0000", 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108822 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s6 41/33 ECSubWrite(tid=199, reqid=osd.13.0:199, at_version=41'20, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.922622+0000", 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.5405655660000002, 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011657239999999999, 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922622+0000", 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922622+0000", 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922623+0000", 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5300000000000003e-07 2024-09-16T04:01:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922670+0000", 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6928999999999997e-05 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922694+0000", 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3969999999999999e-05 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922713+0000", 2024-09-16T04:01:30.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8618999999999999e-05 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.924868+0000", 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021553900000000001 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.924892+0000", 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3952999999999999e-05 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:30.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.934162+0000", 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092705280000000001 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.934279+0000", 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011699799999999999 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:30.573 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:30.575 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:01:30.575 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:30.574+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:30.676 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:30.672+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:30.747 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:30.748 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:30.748 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:01:30.748 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:30.761 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:01:30.777 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:30.774+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:30.878 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:30.872+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:30.879 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:30.879 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:30.880 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:01:30.880 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:01:30.880 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:30.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:01:30.923 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T04:01:30.928 INFO:tasks.rados.rados.0.smithi188.stdout:141: done (0 left) 2024-09-16T04:01:30.928 INFO:tasks.rados.rados.0.smithi188.stdout:142: snap_create 2024-09-16T04:01:30.929 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:01:30.951 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T04:01:30.997 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:30.992+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:31.015 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:31.015 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:01:31.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:01:31.016 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:01:31.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.016 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:106 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~589824,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.016 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.932645+0000", 2024-09-16T04:01:31.016 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0797036749999993, 2024-09-16T04:01:31.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0062299809999999999, 2024-09-16T04:01:31.016 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 106 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.932645+0000", 2024-09-16T04:01:31.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.932645+0000", 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.932646+0000", 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.398e-06 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933166+0000", 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052027300000000002 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933189+0000", 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2221999999999998e-05 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933200+0000", 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1442e-05 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933268+0000", 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7545999999999993e-05 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933413+0000", 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014544699999999999 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.934620+0000", 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012073520000000001 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936195+0000", 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001574443 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.938788+0000", 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025934930000000001 2024-09-16T04:01:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.938875+0000", 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6365000000000004e-05 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:107 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.933223+0000", 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0791251200000005, 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074324209999999998, 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 107 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933223+0000", 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933223+0000", 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933224+0000", 2024-09-16T04:01:31.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1650000000000001e-06 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933229+0000", 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5249999999999999e-06 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933237+0000", 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7279999999999995e-06 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933245+0000", 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6319999999999997e-06 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.935333+0000", 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020876430000000001 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.935414+0000", 2024-09-16T04:01:31.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1515999999999997e-05 2024-09-16T04:01:31.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.935785+0000", 2024-09-16T04:01:31.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037085299999999998 2024-09-16T04:01:31.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936626+0000", 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084103900000000002 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.940589+0000", 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039625709999999998 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.940656+0000", 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6748999999999994e-05 2024-09-16T04:01:31.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:108 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0790926249999995, 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014725059, 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.029 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 108 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2399999999999998e-07 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933259+0000", 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.937e-06 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933267+0000", 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6229999999999997e-06 2024-09-16T04:01:31.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933273+0000", 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936240+0000", 2024-09-16T04:01:31.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002967365 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936330+0000", 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0200999999999999e-05 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.940982+0000", 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046523210000000001 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.941058+0000", 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5798999999999997e-05 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.947981+0000", 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006922615 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/33 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.941991+0000", 2024-09-16T04:01:31.034 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0703572759999993, 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005540227, 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.941991+0000", 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.941991+0000", 2024-09-16T04:01:31.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.941992+0000", 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0399999999999995e-07 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942018+0000", 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6254e-05 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942036+0000", 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8385000000000001e-05 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942044+0000", 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5970000000000003e-06 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.947501+0000", 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054569229999999998 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.947531+0000", 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0363999999999999e-05 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.039 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/33 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-16T04:01:31.039 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.942276+0000", 2024-09-16T04:01:31.039 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0700726510000003, 2024-09-16T04:01:31.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055754760000000002, 2024-09-16T04:01:31.039 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.039 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942276+0000", 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942276+0000", 2024-09-16T04:01:31.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942276+0000", 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9799999999999999e-07 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942287+0000", 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0936e-05 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942295+0000", 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1140000000000006e-06 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.942300+0000", 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9490000000000004e-06 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.947808+0000", 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055076550000000002 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.947851+0000", 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3523999999999999e-05 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.043 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:139 3.5s0 3:a37b04c5:::smithi18831615-47:head [append 0~786432,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.980591+0000", 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0317572980000005, 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066524640000000003, 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 139 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980591+0000", 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980591+0000", 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980593+0000", 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.598e-06 2024-09-16T04:01:31.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980971+0000", 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037804800000000002 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980988+0000", 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7509000000000001e-05 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980997+0000", 2024-09-16T04:01:31.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5660000000000007e-06 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981057+0000", 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9997000000000002e-05 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981176+0000", 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011877799999999999 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.982015+0000", 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00083982700000000002 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.984671+0000", 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026555139999999999 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987160+0000", 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024886560000000001 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987244+0000", 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3971e-05 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:140 3.5s0 3:a37b04c5:::smithi18831615-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.981017+0000", 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0313315139999997, 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006474301, 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.049 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 140 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981017+0000", 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981017+0000", 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981018+0000", 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2799999999999996e-07 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981023+0000", 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4700000000000001e-06 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981031+0000", 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3569999999999998e-06 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981038+0000", 2024-09-16T04:01:31.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2740000000000004e-06 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.982734+0000", 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016962220000000001 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.982816+0000", 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1743999999999995e-05 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.983075+0000", 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000259213 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.984697+0000", 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001622615 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987396+0000", 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026981140000000002 2024-09-16T04:01:31.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987491+0000", 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5663999999999998e-05 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:141 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0313013739999999, 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087077830000000002, 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 141 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.056 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981048+0000", 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981051+0000", 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1659999999999998e-06 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981057+0000", 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6229999999999999e-06 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981063+0000", 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6550000000000001e-06 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.983220+0000", 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021567890000000001 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.983291+0000", 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1326000000000001e-05 2024-09-16T04:01:31.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987840+0000", 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045492309999999999 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987922+0000", 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1205000000000006e-05 2024-09-16T04:01:31.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.989755+0000", 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018333049999999999 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:169 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:22.025583+0000", 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9867658020000007, 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054890010000000003, 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.062 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 169 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.025583+0000", 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.025583+0000", 2024-09-16T04:01:31.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.025584+0000", 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.547e-06 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.025725+0000", 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014079399999999999 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.025743+0000", 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7618e-05 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.025751+0000", 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0630000000000006e-06 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.025806+0000", 2024-09-16T04:01:31.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5152000000000003e-05 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.025883+0000", 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7521999999999994e-05 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.026495+0000", 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061172400000000001 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.027683+0000", 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011882240000000001 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030974+0000", 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032902550000000002 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.031072+0000", 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8102000000000004e-05 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.067 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:171 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:22.028441+0000", 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9839074029999999, 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007731295, 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.068 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 171 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028441+0000", 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028441+0000", 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028442+0000", 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.37e-07 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028445+0000", 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8290000000000001e-06 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028454+0000", 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.371e-06 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028460+0000", 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0440000000000001e-06 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.029650+0000", 2024-09-16T04:01:31.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011897570000000001 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.029706+0000", 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6484000000000003e-05 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030075+0000", 2024-09-16T04:01:31.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036891499999999999 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030774+0000", 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069939000000000004 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036074+0000", 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052994870000000003 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036172+0000", 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8381000000000006e-05 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:172 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9838784290000007, 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091168310000000006, 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.074 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 172 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9900000000000001e-07 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028474+0000", 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6969999999999999e-06 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028482+0000", 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6420000000000007e-06 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028490+0000", 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.712e-06 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030305+0000", 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001815327 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030412+0000", 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107055 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036597+0000", 2024-09-16T04:01:31.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061849360000000003 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036681+0000", 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4110999999999995e-05 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.037587+0000", 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090585199999999996 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.079 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:314 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~491520] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.0017012599999999, 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013929950999999999, 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.080 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 314 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010651+0000", 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.467e-06 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011198+0000", 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054779799999999995 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011237+0000", 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8092000000000001e-05 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011254+0000", 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7232999999999999e-05 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011454+0000", 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020057300000000001 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011623+0000", 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016875899999999999 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014078+0000", 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024548510000000001 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024181+0000", 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010102877 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024366+0000", 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018482199999999999 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024577+0000", 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021147900000000001 2024-09-16T04:01:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.087 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.087 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.087 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:315 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~770048] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:31.087 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.9961034040000003, 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091517409999999997, 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 315 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016247+0000", 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1799999999999999e-06 2024-09-16T04:01:31.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017297+0000", 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010498269999999999 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017327+0000", 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0133000000000001e-05 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017349+0000", 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017547+0000", 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019868199999999999 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017839+0000", 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029153000000000002 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020248+0000", 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024092649999999998 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024592+0000", 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043436849999999999 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025288+0000", 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069595499999999995 2024-09-16T04:01:31.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025397+0000", 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010890799999999999 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:316 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (56) in=7b] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.9949355659999997, 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010644074999999999, 2024-09-16T04:01:31.093 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 316 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017415+0000", 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.99e-06 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017424+0000", 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4129999999999995e-06 2024-09-16T04:01:31.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017444+0000", 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0115999999999998e-05 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017463+0000", 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8199999999999999e-05 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022150+0000", 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046871079999999997 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022414+0000", 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000264491 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023228+0000", 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081356500000000003 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025417+0000", 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021892959999999999 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.027816+0000", 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023989509999999999 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028057+0000", 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000240945 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.098 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:317 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.9948244920000002, 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017770083999999998, 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:31.099 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 317 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017526+0000", 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017533+0000", 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1960000000000002e-06 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017555+0000", 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2157000000000001e-05 2024-09-16T04:01:31.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017574+0000", 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8298999999999999e-05 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024692+0000", 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071183920000000003 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024964+0000", 2024-09-16T04:01:31.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027219700000000001 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029509+0000", 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045448850000000002 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031984+0000", 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024747929999999999 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.035294+0000", 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033101850000000002 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 40/33 ECSubWriteReply(tid=18, last_complete=40'13, committed=1, applied=1))", 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017864+0000", 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.9944844360000005, 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059170250000000002, 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.104 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017864+0000", 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017864+0000", 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017866+0000", 2024-09-16T04:01:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.587e-06 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017874+0000", 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017911+0000", 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6869000000000002e-05 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017925+0000", 2024-09-16T04:01:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3453e-05 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023759+0000", 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058348890000000002 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023781+0000", 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1600999999999999e-05 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.0s2 41/33 ECSubWrite(tid=58, reqid=client.4646.0:330, at_version=41'35, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.910330+0000", 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.1020185759999999, 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086055409999999995, 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910330+0000", 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910330+0000", 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910331+0000", 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7699999999999992e-07 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910381+0000", 2024-09-16T04:01:31.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0118999999999999e-05 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910409+0000", 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7959000000000001e-05 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910429+0000", 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9930000000000001e-05 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911102+0000", 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000673228 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911126+0000", 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3536999999999998e-05 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918848+0000", 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077223389999999999 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918935+0000", 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7452000000000002e-05 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/33 ECSubWrite(tid=197, reqid=osd.13.0:197, at_version=41'18, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.920329+0000", 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.0920190090000004, 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055560490000000004, 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.112 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920329+0000", 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920329+0000", 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920331+0000", 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.843e-06 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920392+0000", 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0338999999999999e-05 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920437+0000", 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5762e-05 2024-09-16T04:01:31.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920462+0000", 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4975999999999999e-05 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920715+0000", 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025296800000000001 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920741+0000", 2024-09-16T04:01:31.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5449e-05 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.925773+0000", 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050320010000000004 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.925885+0000", 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112711 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/33 ECSubWrite(tid=199, reqid=osd.13.0:199, at_version=41'20, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.922825+0000", 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.0895236930000003, 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055295689999999998, 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922825+0000", 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922825+0000", 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922826+0000", 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.226e-06 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922875+0000", 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9206000000000003e-05 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922900+0000", 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4675999999999999e-05 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.922919+0000", 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8689999999999999e-05 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.923069+0000", 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000150786 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.923093+0000", 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4202999999999998e-05 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.928267+0000", 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051731770000000002 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.928354+0000", 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7604999999999999e-05 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.0s2 41/33 ECSubWrite(tid=61, reqid=osd.14.0:60, at_version=0'0, trim_to=0'0, roll_forward_to=41'39))", 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.930471+0000", 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.0818772259999996, 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060718669999999999, 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:31.121 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930471+0000", 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930471+0000", 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930473+0000", 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.829e-06 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930525+0000", 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1693000000000001e-05 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930559+0000", 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4402e-05 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930596+0000", 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6896e-05 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931008+0000", 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041160799999999999 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931032+0000", 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4488999999999999e-05 2024-09-16T04:01:31.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.936440+0000", 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054074300000000004 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.936543+0000", 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010352 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:31.125 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:31.126 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:31.127 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:01:31.128 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:31.122+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:31.228 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:31.222+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:31.245 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:31.245 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:31.245 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:31.245 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:31.259 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:01:31.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:31.264+0000 7f61a54da640 0 --2- 172.21.15.37:0/1977389127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61a00c8c80 0x7f61a01436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:01:31.329 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:31.320+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:31.380 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:31.380 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:31.380 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:31.380 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:31.380 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:31.394 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:01:31.441 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:31.436+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:31.531 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:31.524+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.887532+0000", 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.6690664469999996, 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032828089999999998, 2024-09-16T04:01:31.561 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.887532+0000", 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.887532+0000", 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.887533+0000", 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2800000000000003e-07 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888152+0000", 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00061955799999999998 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888173+0000", 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0939999999999999e-05 2024-09-16T04:01:31.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888200+0000", 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.741e-05 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888290+0000", 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9889999999999995e-05 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888304+0000", 2024-09-16T04:01:31.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3339000000000001e-05 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890767+0000", 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0024633340000000002 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890815+0000", 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8010000000000003e-05 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 39/33 ECSubWrite(tid=81, reqid=client.4646.0:292, at_version=39'42, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.891645+0000", 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.6649534020000001, 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016311679999999999, 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891645+0000", 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891645+0000", 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891647+0000", 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.485e-06 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891743+0000", 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6371000000000003e-05 2024-09-16T04:01:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891762+0000", 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9032999999999999e-05 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891772+0000", 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9850000000000001e-06 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891868+0000", 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5616000000000001e-05 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891880+0000", 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2065999999999999e-05 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893235+0000", 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013552080000000001 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893276+0000", 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1403999999999999e-05 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 39/33 ECSubWrite(tid=82, reqid=client.4646.0:293, at_version=39'43, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.893534+0000", 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.6630649349999995, 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0017716260000000001, 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893534+0000", 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893534+0000", 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893534+0000", 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893551+0000", 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6215999999999999e-05 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893561+0000", 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0609e-05 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893568+0000", 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2489999999999998e-06 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893627+0000", 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9660999999999997e-05 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.893639+0000", 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1691999999999999e-05 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.895267+0000", 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016284649999999999 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.895305+0000", 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7994000000000002e-05 2024-09-16T04:01:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "pg_update_log_missing(3.4s4 epoch 40/33 rep_tid 30 entries 40'14 (0'0) error 3:2e4a2a62:::smithi18831615-31:head by client.4646.0:296 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 36'11)", 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.890363+0000", 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.666236048, 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020761550000000001, 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.890363+0000", 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.890363+0000", 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.890363+0000", 2024-09-16T04:01:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6600000000000002e-07 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.890374+0000", 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1046000000000001e-05 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.890397+0000", 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3187999999999999e-05 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.890417+0000", 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9752e-05 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.890551+0000", 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013353799999999999 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.892439+0000", 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0018882650000000001 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 40/33 ECSubWrite(tid=85, reqid=client.4646.0:297, at_version=40'45, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.905804+0000", 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6507946530000002, 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048358100000000003, 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.905804+0000", 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.905804+0000", 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.905804+0000", 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1999999999999999e-07 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906147+0000", 2024-09-16T04:01:31.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034335700000000003 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906161+0000", 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3538e-05 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906174+0000", 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3137e-05 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906264+0000", 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9747999999999995e-05 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906278+0000", 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3997e-05 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.910597+0000", 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0043189270000000002 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.910640+0000", 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2985999999999998e-05 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 40/33 ECSubWrite(tid=86, reqid=osd.11.0:86, at_version=40'47, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:31.583 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.913114+0000", 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6434848429999995, 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020748820000000001, 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.913114+0000", 2024-09-16T04:01:31.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.913114+0000", 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.913115+0000", 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1519999999999999e-06 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.913141+0000", 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5582999999999999e-05 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.913154+0000", 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3558e-05 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.913162+0000", 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5719999999999997e-06 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.913237+0000", 2024-09-16T04:01:31.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5582999999999998e-05 2024-09-16T04:01:31.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.913249+0000", 2024-09-16T04:01:31.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2089000000000001e-05 2024-09-16T04:01:31.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.915147+0000", 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001897244 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.915189+0000", 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2101000000000002e-05 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.588 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 40/33 ECSubWrite(tid=87, reqid=client.4646.0:298, at_version=40'48, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.916214+0000", 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6403847709999999, 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021390329999999998, 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.916214+0000", 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.916214+0000", 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.916215+0000", 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1799999999999995e-07 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.916562+0000", 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034736199999999999 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.916586+0000", 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.389e-05 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.916594+0000", 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8860000000000004e-06 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.916650+0000", 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6480000000000001e-05 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.916660+0000", 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5279999999999992e-06 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.918323+0000", 2024-09-16T04:01:31.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001662866 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.918353+0000", 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0403e-05 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 40/33 ECSubWrite(tid=88, reqid=client.4646.0:298, at_version=0'0, trim_to=0'0, roll_forward_to=40'48))", 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.920738+0000", 2024-09-16T04:01:31.593 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6358605500000003, 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002420864, 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920738+0000", 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920738+0000", 2024-09-16T04:01:31.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920739+0000", 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0999999999999997e-07 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920761+0000", 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1984000000000002e-05 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920773+0000", 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1603000000000001e-05 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920779+0000", 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6320000000000002e-06 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920826+0000", 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6706999999999999e-05 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920832+0000", 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5759999999999996e-06 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.923099+0000", 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022679240000000002 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.923159+0000", 2024-09-16T04:01:31.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9528000000000003e-05 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 40/33 ECSubWrite(tid=89, reqid=client.4646.0:299, at_version=40'49, trim_to=0'0, roll_forward_to=40'48))", 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.922452+0000", 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6341468319999999, 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015816999999999999, 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.922452+0000", 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.922452+0000", 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.922453+0000", 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8999999999999996e-07 2024-09-16T04:01:31.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.922513+0000", 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0364e-05 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.922524+0000", 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1462e-05 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.922530+0000", 2024-09-16T04:01:31.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9830000000000001e-06 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.922578+0000", 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7364999999999998e-05 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.922586+0000", 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5809999999999997e-06 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.601 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924005+0000", 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00141851 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924034+0000", 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8745e-05 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 40/33 ECSubWrite(tid=90, reqid=client.4646.0:300, at_version=40'50, trim_to=0'0, roll_forward_to=40'48))", 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.924876+0000", 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6317227120000002, 2024-09-16T04:01:31.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081982599999999995, 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924876+0000", 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924876+0000", 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924877+0000", 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5199999999999996e-07 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924892+0000", 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5705e-05 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924906+0000", 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3501e-05 2024-09-16T04:01:31.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924912+0000", 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0719999999999996e-06 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924969+0000", 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7024000000000002e-05 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.924974+0000", 2024-09-16T04:01:31.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2089999999999996e-06 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.925665+0000", 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00069070500000000001 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.925696+0000", 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0858000000000002e-05 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 40/33 ECSubWrite(tid=91, reqid=client.4646.0:303, at_version=40'52, trim_to=0'0, roll_forward_to=40'49))", 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.928788+0000", 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6278111400000004, 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001030216, 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.928788+0000", 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.928788+0000", 2024-09-16T04:01:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.928788+0000", 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7100000000000001e-07 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.928804+0000", 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6251000000000001e-05 2024-09-16T04:01:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.928816+0000", 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.184e-05 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.928825+0000", 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2840000000000006e-06 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.928876+0000", 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1916000000000002e-05 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.928883+0000", 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.489e-06 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.929782+0000", 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00089934200000000002 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.929818+0000", 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5423000000000003e-05 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs6 40/33 ECSubWrite(tid=94, reqid=client.4646.0:309, at_version=40'16, trim_to=0'0, roll_forward_to=37'14))", 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.989938+0000", 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5666610580000002, 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001618429, 2024-09-16T04:01:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.989938+0000", 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.989938+0000", 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.989939+0000", 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0139999999999999e-06 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.990428+0000", 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00048943200000000004 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.990447+0000", 2024-09-16T04:01:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8715e-05 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.990457+0000", 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0703999999999999e-05 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.990548+0000", 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0419000000000003e-05 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.990555+0000", 2024-09-16T04:01:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4569999999999999e-06 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.991522+0000", 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00096650900000000005 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.991556+0000", 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4178999999999999e-05 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs6 40/33 ECSubWrite(tid=95, reqid=client.4646.0:310, at_version=40'17, trim_to=0'0, roll_forward_to=37'14))", 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.995509+0000", 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5610900760000002, 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014598129999999999, 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.995509+0000", 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.995509+0000", 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.995509+0000", 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.995635+0000", 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000125141 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.995649+0000", 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.448e-05 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.995657+0000", 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7409999999999992e-06 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.995739+0000", 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2488e-05 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.995752+0000", 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2611e-05 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.996926+0000", 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0011736229999999999 2024-09-16T04:01:31.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.996968+0000", 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2828999999999999e-05 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs6 40/33 ECSubWrite(tid=96, reqid=client.4646.0:311, at_version=40'18, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.998900+0000", 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5576988009999999, 2024-09-16T04:01:31.621 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00095661100000000001, 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.998900+0000", 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.998900+0000", 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.622 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.998901+0000", 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0599999999999999e-07 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.998947+0000", 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6650000000000002e-05 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.998961+0000", 2024-09-16T04:01:31.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4073999999999999e-05 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.998968+0000", 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1060000000000002e-06 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.999031+0000", 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2267000000000005e-05 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.999045+0000", 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3739e-05 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.999819+0000", 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000774912 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.999856+0000", 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7057000000000001e-05 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.625 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s5 40/33 ECSubWrite(tid=18, reqid=client.4646.0:314, at_version=40'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.012344+0000", 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5442550730000004, 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001883226, 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.012344+0000", 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.626 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.012344+0000", 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.012344+0000", 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1600000000000002e-07 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.627 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.012644+0000", 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030023700000000002 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.012673+0000", 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8813e-05 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.012688+0000", 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4593000000000001e-05 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.012779+0000", 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1352000000000002e-05 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.012789+0000", 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0016e-05 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.014187+0000", 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013976150000000001 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.014227+0000", 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0284000000000003e-05 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s5 40/33 ECSubWrite(tid=19, reqid=client.4646.0:315, at_version=40'14, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.018979+0000", 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5376194319999996, 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020415479999999998, 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.630 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.018979+0000", 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.018979+0000", 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.631 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.018980+0000", 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.133e-06 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.019471+0000", 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00049067000000000002 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.019497+0000", 2024-09-16T04:01:31.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6132e-05 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.019512+0000", 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4375999999999999e-05 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.019603+0000", 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1151999999999997e-05 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.019613+0000", 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0033e-05 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.020981+0000", 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013679320000000001 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.021021+0000", 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0120000000000002e-05 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.634 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s5 40/33 ECSubWrite(tid=20, reqid=client.4646.0:316, at_version=40'15, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.022074+0000", 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5345244820000001, 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001276358, 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022074+0000", 2024-09-16T04:01:31.635 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022074+0000", 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022075+0000", 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0200000000000001e-07 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022091+0000", 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6399999999999999e-05 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022105+0000", 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4076e-05 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022113+0000", 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7570000000000004e-06 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022185+0000", 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2255999999999994e-05 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022195+0000", 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8910000000000007e-06 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.023304+0000", 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001108242 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.023351+0000", 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6934000000000002e-05 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s5 40/33 ECSubWrite(tid=21, reqid=client.4646.0:316, at_version=0'0, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.027301+0000", 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5292977069999996, 2024-09-16T04:01:31.639 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001710874, 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.027301+0000", 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.027301+0000", 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.027302+0000", 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.139e-06 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.027316+0000", 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3759e-05 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.027334+0000", 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8360999999999999e-05 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.027347+0000", 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2404e-05 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.027428+0000", 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1722000000000001e-05 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.027438+0000", 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5629999999999998e-06 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.028984+0000", 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001545675 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.029012+0000", 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8251000000000001e-05 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 44/33 ECSubWrite(tid=82, reqid=client.4646.0:352, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:29.927832+0000", 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.6287667809999999, 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048677060000000003, 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.644 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.927832+0000", 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.927832+0000", 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.927832+0000", 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3999999999999997e-07 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.927876+0000", 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3599e-05 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.927902+0000", 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5919e-05 2024-09-16T04:01:31.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.927919+0000", 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7289999999999999e-05 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928042+0000", 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000123127 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928061+0000", 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9196999999999999e-05 2024-09-16T04:01:31.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.932643+0000", 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045815220000000002 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.932700+0000", 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6712000000000002e-05 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 44/33 ECSubWrite(tid=83, reqid=client.4646.0:352, at_version=0'0, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:29.933978+0000", 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.6226208610000001, 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015826390000000001, 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933978+0000", 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933978+0000", 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933979+0000", 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.336e-06 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933996+0000", 2024-09-16T04:01:31.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6795000000000001e-05 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934010+0000", 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3757999999999999e-05 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934019+0000", 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4890000000000001e-06 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934144+0000", 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012519699999999999 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934168+0000", 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3662000000000001e-05 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.935528+0000", 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013604069999999999 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:31.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:31.653 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.935560+0000", 2024-09-16T04:01:31.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1995000000000001e-05 2024-09-16T04:01:31.653 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.653 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.653 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.653 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:31.653 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:31.653 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:31.654 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:31.655 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:31.654+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:31.656 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:01:31.710 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T04:01:31.710 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-09-16T04:01:31.715 INFO:tasks.rados.rados.0.smithi188.stdout:142: done (0 left) 2024-09-16T04:01:31.716 INFO:tasks.rados.rados.0.smithi188.stdout:143: snap_create 2024-09-16T04:01:31.740 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T04:01:31.740 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:31.785 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:31.780+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:31.826 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:31.826 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:31.842 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:01:31.845 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:31.857 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:31.854+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:31.871 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:31.891 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:31.891 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:31.913 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:31.913 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:31.933 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:31.933 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:31.952 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:31.953 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:31.958 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:31.952+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:31.962 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:31.963 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:31.973 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:31.976 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:01:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:32.010 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:32.010 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:32.030 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:32.030 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:32.050 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:32.059 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:32.058+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:32.070 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:32.070 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:32.086 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:32.086 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:32.102 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:32.102 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:32.106 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:32.106 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:32.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:32.100+0000 7ff2b0e98640 0 -- 172.21.15.37:0/1703663981 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7ff2ac223f60 msgr2=0x7ff2ac226350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:32.115 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:01:32.159 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:32.154+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:32.196 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:32.197 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:32.215 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2024-09-16T04:01:31.713467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37241395,"num_objects":52,"num_object_clones":5,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":1,"num_read":330,"num_read_kb":28571,"num_write":558,"num_write_kb":48774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":469,"ondisk_log_size":469,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":528048,"kb_used_data":74744,"kb_used_omap":24,"kb_used_meta":452903,"kb_avail":1509421392,"statfs":{"total":1546188226560,"available":1545647505408,"internally_reserved":0,"allocated":76537856,"data_stored":81987299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":463772753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-147456,"num_objects":-17,"num_object_clones":-15,"num_object_copies":-119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-17,"num_whiteouts":-1,"num_read":10,"num_read_kb":6100,"num_write":18,"num_write_kb":5360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.815880"},"pg_stats":[{"pgid":"3.f","version":"42'25","reported_seq":87,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:30.716295+0000","last_change":"2024-09-16T04:01:27.918828+0000","last_active":"2024-09-16T04:01:30.716295+0000","last_peered":"2024-09-16T04:01:30.716295+0000","last_clean":"2024-09-16T04:01:30.716295+0000","last_became_active":"2024-09-16T04:01:19.845422+0000","last_became_peered":"2024-09-16T04:01:19.845422+0000","last_unstale":"2024-09-16T04:01:30.716295+0000","last_undegraded":"2024-09-16T04:01:30.716295+0000","last_fullsized":"2024-09-16T04:01:30.716295+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:44.273275+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.014975014,"stat_sum":{"num_bytes":2752512,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":6103,"num_write":36,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.e","version":"36'5","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:01:27.901622+0000","last_change":"2024-09-16T04:01:27.901963+0000","last_active":"2024-09-16T04:01:27.901622+0000","last_peered":"2024-09-16T04:01:27.901622+0000","last_clean":"2024-09-16T04:01:27.901622+0000","last_became_active":"2024-09-16T04:01:20.266249+0000","last_became_peered":"2024-09-16T04:01:20.266249+0000","last_unstale":"2024-09-16T04:01:27.901622+0000","last_undegraded":"2024-09-16T04:01:27.901622+0000","last_fullsized":"2024-09-16T04:01:27.901622+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:25.220548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055872799999999998,"stat_sum":{"num_bytes":376832,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":2212,"num_write":5,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.d","version":"41'19","reported_seq":70,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.918506+0000","last_change":"2024-09-16T04:01:26.920966+0000","last_active":"2024-09-16T04:01:26.918506+0000","last_peered":"2024-09-16T04:01:26.918506+0000","last_clean":"2024-09-16T04:01:26.918506+0000","last_became_active":"2024-09-16T04:01:19.845009+0000","last_became_peered":"2024-09-16T04:01:19.845009+0000","last_unstale":"2024-09-16T04:01:26.918506+0000","last_undegraded":"2024-09-16T04:01:26.918506+0000","last_fullsized":"2024-09-16T04:01:26.918506+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:44.872962+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.017215513000000002,"stat_sum":{"num_bytes":2523136,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":3462,"num_write":31,"num_write_kb":3600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.c","version":"42'68","reported_seq":129,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:30.716294+0000","last_change":"2024-09-16T04:01:27.935040+0000","last_active":"2024-09-16T04:01:30.716294+0000","last_peered":"2024-09-16T04:01:30.716294+0000","last_clean":"2024-09-16T04:01:30.716294+0000","last_became_active":"2024-09-16T04:01:20.259789+0000","last_became_peered":"2024-09-16T04:01:20.259789+0000","last_unstale":"2024-09-16T04:01:30.716294+0000","last_undegraded":"2024-09-16T04:01:30.716294+0000","last_fullsized":"2024-09-16T04:01:30.716294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:48.131441+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031418335999999998,"stat_sum":{"num_bytes":6176768,"num_objects":6,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":1021,"num_write":58,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.b","version":"44'26","reported_seq":98,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.925472+0000","last_change":"2024-09-16T04:01:27.896225+0000","last_active":"2024-09-16T04:01:29.925472+0000","last_peered":"2024-09-16T04:01:29.925472+0000","last_clean":"2024-09-16T04:01:29.925472+0000","last_became_active":"2024-09-16T04:01:20.268548+0000","last_became_peered":"2024-09-16T04:01:20.268548+0000","last_unstale":"2024-09-16T04:01:29.925472+0000","last_undegraded":"2024-09-16T04:01:29.925472+0000","last_fullsized":"2024-09-16T04:01:29.925472+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883792,"stat_sum":{"num_bytes":3227648,"num_objects":7,"num_object_clones":1,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":29,"num_read_kb":4571,"num_write":41,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.a","version":"41'8","reported_seq":54,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.930755+0000","last_change":"2024-09-16T04:01:26.932768+0000","last_active":"2024-09-16T04:01:26.930755+0000","last_peered":"2024-09-16T04:01:26.930755+0000","last_clean":"2024-09-16T04:01:26.930755+0000","last_became_active":"2024-09-16T04:01:20.268001+0000","last_became_peered":"2024-09-16T04:01:20.268001+0000","last_unstale":"2024-09-16T04:01:26.930755+0000","last_undegraded":"2024-09-16T04:01:26.930755+0000","last_fullsized":"2024-09-16T04:01:26.930755+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:19.820691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019244486000000002,"stat_sum":{"num_bytes":704512,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2066,"num_write":4,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.9","version":"36'4","reported_seq":57,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:01:27.907520+0000","last_change":"2024-09-16T04:01:27.910094+0000","last_active":"2024-09-16T04:01:27.907520+0000","last_peered":"2024-09-16T04:01:27.907520+0000","last_clean":"2024-09-16T04:01:27.907520+0000","last_became_active":"2024-09-16T04:01:20.153728+0000","last_became_peered":"2024-09-16T04:01:20.153728+0000","last_unstale":"2024-09-16T04:01:27.907520+0000","last_undegraded":"2024-09-16T04:01:27.907520+0000","last_fullsized":"2024-09-16T04:01:27.907520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:28.903218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002897096,"stat_sum":{"num_bytes":1048576,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.8","version":"41'8","reported_seq":52,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.937668+0000","last_change":"2024-09-16T04:01:26.939002+0000","last_active":"2024-09-16T04:01:26.937668+0000","last_peered":"2024-09-16T04:01:26.937668+0000","last_clean":"2024-09-16T04:01:26.937668+0000","last_became_active":"2024-09-16T04:01:20.259482+0000","last_became_peered":"2024-09-16T04:01:20.259482+0000","last_unstale":"2024-09-16T04:01:26.937668+0000","last_undegraded":"2024-09-16T04:01:26.937668+0000","last_fullsized":"2024-09-16T04:01:26.937668+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:40.894861+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.041084041000000002,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":238,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.912505+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:26.912505+0000","last_peered":"2024-09-16T04:01:26.912505+0000","last_clean":"2024-09-16T04:01:26.912505+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:26.912505+0000","last_undegraded":"2024-09-16T04:01:26.912505+0000","last_fullsized":"2024-09-16T04:01:26.912505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":41,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.898073+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:26.898073+0000","last_peered":"2024-09-16T04:01:26.898073+0000","last_clean":"2024-09-16T04:01:26.898073+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:26.898073+0000","last_undegraded":"2024-09-16T04:01:26.898073+0000","last_fullsized":"2024-09-16T04:01:26.898073+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"41'20","reported_seq":63,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.938336+0000","last_change":"2024-09-16T04:01:26.940073+0000","last_active":"2024-09-16T04:01:26.938336+0000","last_peered":"2024-09-16T04:01:26.938336+0000","last_clean":"2024-09-16T04:01:26.938336+0000","last_became_active":"2024-09-16T04:01:20.258661+0000","last_became_peered":"2024-09-16T04:01:20.258661+0000","last_unstale":"2024-09-16T04:01:26.938336+0000","last_undegraded":"2024-09-16T04:01:26.938336+0000","last_fullsized":"2024-09-16T04:01:26.938336+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:45.696883+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.026840112999999999,"stat_sum":{"num_bytes":2342912,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":13,"num_write_kb":2864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.912356+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:26.912356+0000","last_peered":"2024-09-16T04:01:26.912356+0000","last_clean":"2024-09-16T04:01:26.912356+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:26.912356+0000","last_undegraded":"2024-09-16T04:01:26.912356+0000","last_fullsized":"2024-09-16T04:01:26.912356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"42'14","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.925154+0000","last_change":"2024-09-16T04:01:27.912978+0000","last_active":"2024-09-16T04:01:29.925154+0000","last_peered":"2024-09-16T04:01:29.925154+0000","last_clean":"2024-09-16T04:01:29.925154+0000","last_became_active":"2024-09-16T04:01:20.269891+0000","last_became_peered":"2024-09-16T04:01:20.269891+0000","last_unstale":"2024-09-16T04:01:29.925154+0000","last_undegraded":"2024-09-16T04:01:29.925154+0000","last_fullsized":"2024-09-16T04:01:29.925154+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:31.741361+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017528344000000001,"stat_sum":{"num_bytes":1015808,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1539,"num_write":44,"num_write_kb":992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":41,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.913934+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:26.913934+0000","last_peered":"2024-09-16T04:01:26.913934+0000","last_clean":"2024-09-16T04:01:26.913934+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:26.913934+0000","last_undegraded":"2024-09-16T04:01:26.913934+0000","last_fullsized":"2024-09-16T04:01:26.913934+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"42'47","reported_seq":101,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:01:27.937547+0000","last_change":"2024-09-16T04:01:27.939028+0000","last_active":"2024-09-16T04:01:27.937547+0000","last_peered":"2024-09-16T04:01:27.937547+0000","last_clean":"2024-09-16T04:01:27.937547+0000","last_became_active":"2024-09-16T04:01:20.257526+0000","last_became_peered":"2024-09-16T04:01:20.257526+0000","last_unstale":"2024-09-16T04:01:27.937547+0000","last_undegraded":"2024-09-16T04:01:27.937547+0000","last_fullsized":"2024-09-16T04:01:27.937547+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:19.330544+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031666692000000003,"stat_sum":{"num_bytes":3063808,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":8,"num_write":48,"num_write_kb":6688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.2","version":"30'2","reported_seq":42,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.913721+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:26.913721+0000","last_peered":"2024-09-16T04:01:26.913721+0000","last_clean":"2024-09-16T04:01:26.913721+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:26.913721+0000","last_undegraded":"2024-09-16T04:01:26.913721+0000","last_fullsized":"2024-09-16T04:01:26.913721+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"41'13","reported_seq":59,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.928193+0000","last_change":"2024-09-16T04:01:26.929609+0000","last_active":"2024-09-16T04:01:26.928193+0000","last_peered":"2024-09-16T04:01:26.928193+0000","last_clean":"2024-09-16T04:01:26.928193+0000","last_became_active":"2024-09-16T04:01:19.840099+0000","last_became_peered":"2024-09-16T04:01:19.840099+0000","last_unstale":"2024-09-16T04:01:26.928193+0000","last_undegraded":"2024-09-16T04:01:26.928193+0000","last_fullsized":"2024-09-16T04:01:26.928193+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:45.322409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016445624999999998,"stat_sum":{"num_bytes":2883584,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":29,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.5","version":"42'21","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:30.326940+0000","last_change":"2024-09-16T04:01:27.940341+0000","last_active":"2024-09-16T04:01:30.326940+0000","last_peered":"2024-09-16T04:01:30.326940+0000","last_clean":"2024-09-16T04:01:30.326940+0000","last_became_active":"2024-09-16T04:01:20.258954+0000","last_became_peered":"2024-09-16T04:01:20.258954+0000","last_unstale":"2024-09-16T04:01:30.326940+0000","last_undegraded":"2024-09-16T04:01:30.326940+0000","last_fullsized":"2024-09-16T04:01:30.326940+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:47.678549+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.020452483,"stat_sum":{"num_bytes":4620288,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":13,"num_write_kb":4512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":46,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.924948+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:29.924948+0000","last_peered":"2024-09-16T04:01:29.924948+0000","last_clean":"2024-09-16T04:01:29.924948+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:29.924948+0000","last_undegraded":"2024-09-16T04:01:29.924948+0000","last_fullsized":"2024-09-16T04:01:29.924948+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"42'22","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.916678+0000","last_change":"2024-09-16T04:01:27.925534+0000","last_active":"2024-09-16T04:01:29.916678+0000","last_peered":"2024-09-16T04:01:29.916678+0000","last_clean":"2024-09-16T04:01:29.916678+0000","last_became_active":"2024-09-16T04:01:20.154175+0000","last_became_peered":"2024-09-16T04:01:20.154175+0000","last_unstale":"2024-09-16T04:01:29.916678+0000","last_undegraded":"2024-09-16T04:01:29.916678+0000","last_fullsized":"2024-09-16T04:01:29.916678+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:47.341331+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.017356626,"stat_sum":{"num_bytes":2670592,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":4231,"num_write":13,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:01:27.895481+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:27.895481+0000","last_peered":"2024-09-16T04:01:27.895481+0000","last_clean":"2024-09-16T04:01:27.895481+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:27.895481+0000","last_undegraded":"2024-09-16T04:01:27.895481+0000","last_fullsized":"2024-09-16T04:01:27.895481+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"36'7","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.925038+0000","last_change":"2024-09-16T04:01:27.896911+0000","last_active":"2024-09-16T04:01:29.925038+0000","last_peered":"2024-09-16T04:01:29.925038+0000","last_clean":"2024-09-16T04:01:29.925038+0000","last_became_active":"2024-09-16T04:01:20.266522+0000","last_became_peered":"2024-09-16T04:01:20.266522+0000","last_unstale":"2024-09-16T04:01:29.925038+0000","last_undegraded":"2024-09-16T04:01:29.925038+0000","last_fullsized":"2024-09-16T04:01:29.925038+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:40.046637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014563709999999999,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":724,"num_write":7,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":46,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.916586+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:29.916586+0000","last_peered":"2024-09-16T04:01:29.916586+0000","last_clean":"2024-09-16T04:01:29.916586+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:29.916586+0000","last_undegraded":"2024-09-16T04:01:29.916586+0000","last_fullsized":"2024-09-16T04:01:29.916586+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"36'2","reported_seq":52,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.901535+0000","last_change":"2024-09-16T04:01:26.902799+0000","last_active":"2024-09-16T04:01:26.901535+0000","last_peered":"2024-09-16T04:01:26.901535+0000","last_clean":"2024-09-16T04:01:26.901535+0000","last_became_active":"2024-09-16T04:01:20.267489+0000","last_became_peered":"2024-09-16T04:01:20.267489+0000","last_unstale":"2024-09-16T04:01:26.901535+0000","last_undegraded":"2024-09-16T04:01:26.901535+0000","last_fullsized":"2024-09-16T04:01:26.901535+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.358111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00150011,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2499,"num_write":2,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:30.326633+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:30.326633+0000","last_peered":"2024-09-16T04:01:30.326633+0000","last_clean":"2024-09-16T04:01:30.326633+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:30.326633+0000","last_undegraded":"2024-09-16T04:01:30.326633+0000","last_fullsized":"2024-09-16T04:01:30.326633+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":36651008,"num_objects":48,"num_object_clones":5,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":1,"num_read":194,"num_read_kb":28453,"num_write":353,"num_write_kb":45312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65142784,"data_stored":70402048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":4468736,"data_stored":5226633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215114,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31076,"kb_used_data":4044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340764,"statfs":{"total":96636764160,"available":96604942336,"internally_reserved":0,"allocated":4141056,"data_stored":4833417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215115,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33316,"kb_used_data":6284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":6434816,"data_stored":7385225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":4480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":4587520,"data_stored":4309221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280524,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":5732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":5869568,"data_stored":5597404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":5940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":6082560,"data_stored":5810396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29452,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342388,"statfs":{"total":96636764160,"available":96606605312,"internally_reserved":0,"allocated":2478080,"data_stored":2205916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":4280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":4382720,"data_stored":4110556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30228,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341612,"statfs":{"total":96636764160,"available":96605810688,"internally_reserved":0,"allocated":3276800,"data_stored":3000559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":4424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":4530176,"data_stored":4258012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36276,"kb_used_data":5088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335564,"statfs":{"total":96636764160,"available":96599617536,"internally_reserved":0,"allocated":5210112,"data_stored":4946666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38240,"kb_used_data":7048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333600,"statfs":{"total":96636764160,"available":96597606400,"internally_reserved":0,"allocated":7217152,"data_stored":8151177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247818,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36500,"kb_used_data":5304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335340,"statfs":{"total":96636764160,"available":96599388160,"internally_reserved":0,"allocated":5431296,"data_stored":6337193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247818,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30120,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341720,"statfs":{"total":96636764160,"available":96605921280,"internally_reserved":0,"allocated":3162112,"data_stored":4102219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36560,"kb_used_data":5372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335280,"statfs":{"total":96636764160,"available":96599326720,"internally_reserved":0,"allocated":5500928,"data_stored":7047224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30708,"kb_used_data":3676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341132,"statfs":{"total":96636764160,"available":96605319168,"internally_reserved":0,"allocated":3764224,"data_stored":4665481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6717440,"data_stored":7905280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4022272,"data_stored":4022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4161536,"data_stored":4161536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":3424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5320704,"data_stored":5320704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4423680,"data_stored":5128192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5500928,"data_stored":5500928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3264512,"data_stored":4419584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5533696,"data_stored":5533696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1929216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3379200,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3833856,"data_stored":3833856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2723840,"data_stored":2723840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4399104,"data_stored":5562368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":3575808,"data_stored":3575808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":3981312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:32.217 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:32.260 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:32.254+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:32.271 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:32.271 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:32.285 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:01:32.331 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:32.352 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:32.352 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:32.361 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:32.358+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:32.445 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:32.445 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:32.458 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:01:32.505 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:32.500+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:32.561 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:32.563 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:32.558+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:32.578 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2024-09-16T04:01:31.713467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37241395,"num_objects":52,"num_object_clones":5,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":1,"num_read":330,"num_read_kb":28571,"num_write":558,"num_write_kb":48774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":469,"ondisk_log_size":469,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":528048,"kb_used_data":74744,"kb_used_omap":24,"kb_used_meta":452903,"kb_avail":1509421392,"statfs":{"total":1546188226560,"available":1545647505408,"internally_reserved":0,"allocated":76537856,"data_stored":81987299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":463772753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-147456,"num_objects":-17,"num_object_clones":-15,"num_object_copies":-119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-17,"num_whiteouts":-1,"num_read":10,"num_read_kb":6100,"num_write":18,"num_write_kb":5360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.815880"},"pg_stats":[{"pgid":"3.f","version":"42'25","reported_seq":87,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:30.716295+0000","last_change":"2024-09-16T04:01:27.918828+0000","last_active":"2024-09-16T04:01:30.716295+0000","last_peered":"2024-09-16T04:01:30.716295+0000","last_clean":"2024-09-16T04:01:30.716295+0000","last_became_active":"2024-09-16T04:01:19.845422+0000","last_became_peered":"2024-09-16T04:01:19.845422+0000","last_unstale":"2024-09-16T04:01:30.716295+0000","last_undegraded":"2024-09-16T04:01:30.716295+0000","last_fullsized":"2024-09-16T04:01:30.716295+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:44.273275+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.014975014,"stat_sum":{"num_bytes":2752512,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":6103,"num_write":36,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.e","version":"36'5","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:01:27.901622+0000","last_change":"2024-09-16T04:01:27.901963+0000","last_active":"2024-09-16T04:01:27.901622+0000","last_peered":"2024-09-16T04:01:27.901622+0000","last_clean":"2024-09-16T04:01:27.901622+0000","last_became_active":"2024-09-16T04:01:20.266249+0000","last_became_peered":"2024-09-16T04:01:20.266249+0000","last_unstale":"2024-09-16T04:01:27.901622+0000","last_undegraded":"2024-09-16T04:01:27.901622+0000","last_fullsized":"2024-09-16T04:01:27.901622+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:25.220548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055872799999999998,"stat_sum":{"num_bytes":376832,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":2212,"num_write":5,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.d","version":"41'19","reported_seq":70,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.918506+0000","last_change":"2024-09-16T04:01:26.920966+0000","last_active":"2024-09-16T04:01:26.918506+0000","last_peered":"2024-09-16T04:01:26.918506+0000","last_clean":"2024-09-16T04:01:26.918506+0000","last_became_active":"2024-09-16T04:01:19.845009+0000","last_became_peered":"2024-09-16T04:01:19.845009+0000","last_unstale":"2024-09-16T04:01:26.918506+0000","last_undegraded":"2024-09-16T04:01:26.918506+0000","last_fullsized":"2024-09-16T04:01:26.918506+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:44.872962+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.017215513000000002,"stat_sum":{"num_bytes":2523136,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":3462,"num_write":31,"num_write_kb":3600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.c","version":"42'68","reported_seq":129,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:30.716294+0000","last_change":"2024-09-16T04:01:27.935040+0000","last_active":"2024-09-16T04:01:30.716294+0000","last_peered":"2024-09-16T04:01:30.716294+0000","last_clean":"2024-09-16T04:01:30.716294+0000","last_became_active":"2024-09-16T04:01:20.259789+0000","last_became_peered":"2024-09-16T04:01:20.259789+0000","last_unstale":"2024-09-16T04:01:30.716294+0000","last_undegraded":"2024-09-16T04:01:30.716294+0000","last_fullsized":"2024-09-16T04:01:30.716294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:48.131441+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031418335999999998,"stat_sum":{"num_bytes":6176768,"num_objects":6,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":1021,"num_write":58,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.b","version":"44'26","reported_seq":98,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.925472+0000","last_change":"2024-09-16T04:01:27.896225+0000","last_active":"2024-09-16T04:01:29.925472+0000","last_peered":"2024-09-16T04:01:29.925472+0000","last_clean":"2024-09-16T04:01:29.925472+0000","last_became_active":"2024-09-16T04:01:20.268548+0000","last_became_peered":"2024-09-16T04:01:20.268548+0000","last_unstale":"2024-09-16T04:01:29.925472+0000","last_undegraded":"2024-09-16T04:01:29.925472+0000","last_fullsized":"2024-09-16T04:01:29.925472+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883792,"stat_sum":{"num_bytes":3227648,"num_objects":7,"num_object_clones":1,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":29,"num_read_kb":4571,"num_write":41,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.a","version":"41'8","reported_seq":54,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.930755+0000","last_change":"2024-09-16T04:01:26.932768+0000","last_active":"2024-09-16T04:01:26.930755+0000","last_peered":"2024-09-16T04:01:26.930755+0000","last_clean":"2024-09-16T04:01:26.930755+0000","last_became_active":"2024-09-16T04:01:20.268001+0000","last_became_peered":"2024-09-16T04:01:20.268001+0000","last_unstale":"2024-09-16T04:01:26.930755+0000","last_undegraded":"2024-09-16T04:01:26.930755+0000","last_fullsized":"2024-09-16T04:01:26.930755+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:19.820691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019244486000000002,"stat_sum":{"num_bytes":704512,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2066,"num_write":4,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.9","version":"36'4","reported_seq":57,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:01:27.907520+0000","last_change":"2024-09-16T04:01:27.910094+0000","last_active":"2024-09-16T04:01:27.907520+0000","last_peered":"2024-09-16T04:01:27.907520+0000","last_clean":"2024-09-16T04:01:27.907520+0000","last_became_active":"2024-09-16T04:01:20.153728+0000","last_became_peered":"2024-09-16T04:01:20.153728+0000","last_unstale":"2024-09-16T04:01:27.907520+0000","last_undegraded":"2024-09-16T04:01:27.907520+0000","last_fullsized":"2024-09-16T04:01:27.907520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:28.903218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002897096,"stat_sum":{"num_bytes":1048576,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.8","version":"41'8","reported_seq":52,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.937668+0000","last_change":"2024-09-16T04:01:26.939002+0000","last_active":"2024-09-16T04:01:26.937668+0000","last_peered":"2024-09-16T04:01:26.937668+0000","last_clean":"2024-09-16T04:01:26.937668+0000","last_became_active":"2024-09-16T04:01:20.259482+0000","last_became_peered":"2024-09-16T04:01:20.259482+0000","last_unstale":"2024-09-16T04:01:26.937668+0000","last_undegraded":"2024-09-16T04:01:26.937668+0000","last_fullsized":"2024-09-16T04:01:26.937668+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:40.894861+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.041084041000000002,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":238,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.912505+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:26.912505+0000","last_peered":"2024-09-16T04:01:26.912505+0000","last_clean":"2024-09-16T04:01:26.912505+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:26.912505+0000","last_undegraded":"2024-09-16T04:01:26.912505+0000","last_fullsized":"2024-09-16T04:01:26.912505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":41,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.898073+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:26.898073+0000","last_peered":"2024-09-16T04:01:26.898073+0000","last_clean":"2024-09-16T04:01:26.898073+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:26.898073+0000","last_undegraded":"2024-09-16T04:01:26.898073+0000","last_fullsized":"2024-09-16T04:01:26.898073+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"41'20","reported_seq":63,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.938336+0000","last_change":"2024-09-16T04:01:26.940073+0000","last_active":"2024-09-16T04:01:26.938336+0000","last_peered":"2024-09-16T04:01:26.938336+0000","last_clean":"2024-09-16T04:01:26.938336+0000","last_became_active":"2024-09-16T04:01:20.258661+0000","last_became_peered":"2024-09-16T04:01:20.258661+0000","last_unstale":"2024-09-16T04:01:26.938336+0000","last_undegraded":"2024-09-16T04:01:26.938336+0000","last_fullsized":"2024-09-16T04:01:26.938336+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:45.696883+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.026840112999999999,"stat_sum":{"num_bytes":2342912,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":13,"num_write_kb":2864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.912356+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:26.912356+0000","last_peered":"2024-09-16T04:01:26.912356+0000","last_clean":"2024-09-16T04:01:26.912356+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:26.912356+0000","last_undegraded":"2024-09-16T04:01:26.912356+0000","last_fullsized":"2024-09-16T04:01:26.912356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"42'14","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.925154+0000","last_change":"2024-09-16T04:01:27.912978+0000","last_active":"2024-09-16T04:01:29.925154+0000","last_peered":"2024-09-16T04:01:29.925154+0000","last_clean":"2024-09-16T04:01:29.925154+0000","last_became_active":"2024-09-16T04:01:20.269891+0000","last_became_peered":"2024-09-16T04:01:20.269891+0000","last_unstale":"2024-09-16T04:01:29.925154+0000","last_undegraded":"2024-09-16T04:01:29.925154+0000","last_fullsized":"2024-09-16T04:01:29.925154+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:31.741361+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017528344000000001,"stat_sum":{"num_bytes":1015808,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1539,"num_write":44,"num_write_kb":992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":41,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.913934+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:26.913934+0000","last_peered":"2024-09-16T04:01:26.913934+0000","last_clean":"2024-09-16T04:01:26.913934+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:26.913934+0000","last_undegraded":"2024-09-16T04:01:26.913934+0000","last_fullsized":"2024-09-16T04:01:26.913934+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"42'47","reported_seq":101,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:01:27.937547+0000","last_change":"2024-09-16T04:01:27.939028+0000","last_active":"2024-09-16T04:01:27.937547+0000","last_peered":"2024-09-16T04:01:27.937547+0000","last_clean":"2024-09-16T04:01:27.937547+0000","last_became_active":"2024-09-16T04:01:20.257526+0000","last_became_peered":"2024-09-16T04:01:20.257526+0000","last_unstale":"2024-09-16T04:01:27.937547+0000","last_undegraded":"2024-09-16T04:01:27.937547+0000","last_fullsized":"2024-09-16T04:01:27.937547+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:19.330544+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031666692000000003,"stat_sum":{"num_bytes":3063808,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":8,"num_write":48,"num_write_kb":6688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.2","version":"30'2","reported_seq":42,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.913721+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:26.913721+0000","last_peered":"2024-09-16T04:01:26.913721+0000","last_clean":"2024-09-16T04:01:26.913721+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:26.913721+0000","last_undegraded":"2024-09-16T04:01:26.913721+0000","last_fullsized":"2024-09-16T04:01:26.913721+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"41'13","reported_seq":59,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.928193+0000","last_change":"2024-09-16T04:01:26.929609+0000","last_active":"2024-09-16T04:01:26.928193+0000","last_peered":"2024-09-16T04:01:26.928193+0000","last_clean":"2024-09-16T04:01:26.928193+0000","last_became_active":"2024-09-16T04:01:19.840099+0000","last_became_peered":"2024-09-16T04:01:19.840099+0000","last_unstale":"2024-09-16T04:01:26.928193+0000","last_undegraded":"2024-09-16T04:01:26.928193+0000","last_fullsized":"2024-09-16T04:01:26.928193+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:45.322409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016445624999999998,"stat_sum":{"num_bytes":2883584,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":29,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.5","version":"42'21","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:30.326940+0000","last_change":"2024-09-16T04:01:27.940341+0000","last_active":"2024-09-16T04:01:30.326940+0000","last_peered":"2024-09-16T04:01:30.326940+0000","last_clean":"2024-09-16T04:01:30.326940+0000","last_became_active":"2024-09-16T04:01:20.258954+0000","last_became_peered":"2024-09-16T04:01:20.258954+0000","last_unstale":"2024-09-16T04:01:30.326940+0000","last_undegraded":"2024-09-16T04:01:30.326940+0000","last_fullsized":"2024-09-16T04:01:30.326940+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:47.678549+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.020452483,"stat_sum":{"num_bytes":4620288,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":13,"num_write_kb":4512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":46,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.924948+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:29.924948+0000","last_peered":"2024-09-16T04:01:29.924948+0000","last_clean":"2024-09-16T04:01:29.924948+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:29.924948+0000","last_undegraded":"2024-09-16T04:01:29.924948+0000","last_fullsized":"2024-09-16T04:01:29.924948+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"42'22","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.916678+0000","last_change":"2024-09-16T04:01:27.925534+0000","last_active":"2024-09-16T04:01:29.916678+0000","last_peered":"2024-09-16T04:01:29.916678+0000","last_clean":"2024-09-16T04:01:29.916678+0000","last_became_active":"2024-09-16T04:01:20.154175+0000","last_became_peered":"2024-09-16T04:01:20.154175+0000","last_unstale":"2024-09-16T04:01:29.916678+0000","last_undegraded":"2024-09-16T04:01:29.916678+0000","last_fullsized":"2024-09-16T04:01:29.916678+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:47.341331+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.017356626,"stat_sum":{"num_bytes":2670592,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":4231,"num_write":13,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:01:27.895481+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:27.895481+0000","last_peered":"2024-09-16T04:01:27.895481+0000","last_clean":"2024-09-16T04:01:27.895481+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:27.895481+0000","last_undegraded":"2024-09-16T04:01:27.895481+0000","last_fullsized":"2024-09-16T04:01:27.895481+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"36'7","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.925038+0000","last_change":"2024-09-16T04:01:27.896911+0000","last_active":"2024-09-16T04:01:29.925038+0000","last_peered":"2024-09-16T04:01:29.925038+0000","last_clean":"2024-09-16T04:01:29.925038+0000","last_became_active":"2024-09-16T04:01:20.266522+0000","last_became_peered":"2024-09-16T04:01:20.266522+0000","last_unstale":"2024-09-16T04:01:29.925038+0000","last_undegraded":"2024-09-16T04:01:29.925038+0000","last_fullsized":"2024-09-16T04:01:29.925038+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:40.046637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014563709999999999,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":724,"num_write":7,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":46,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:29.916586+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:29.916586+0000","last_peered":"2024-09-16T04:01:29.916586+0000","last_clean":"2024-09-16T04:01:29.916586+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:29.916586+0000","last_undegraded":"2024-09-16T04:01:29.916586+0000","last_fullsized":"2024-09-16T04:01:29.916586+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"36'2","reported_seq":52,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:01:26.901535+0000","last_change":"2024-09-16T04:01:26.902799+0000","last_active":"2024-09-16T04:01:26.901535+0000","last_peered":"2024-09-16T04:01:26.901535+0000","last_clean":"2024-09-16T04:01:26.901535+0000","last_became_active":"2024-09-16T04:01:20.267489+0000","last_became_peered":"2024-09-16T04:01:20.267489+0000","last_unstale":"2024-09-16T04:01:26.901535+0000","last_undegraded":"2024-09-16T04:01:26.901535+0000","last_fullsized":"2024-09-16T04:01:26.901535+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.358111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00150011,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2499,"num_write":2,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:01:30.326633+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:30.326633+0000","last_peered":"2024-09-16T04:01:30.326633+0000","last_clean":"2024-09-16T04:01:30.326633+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:30.326633+0000","last_undegraded":"2024-09-16T04:01:30.326633+0000","last_fullsized":"2024-09-16T04:01:30.326633+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":36651008,"num_objects":48,"num_object_clones":5,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":1,"num_read":194,"num_read_kb":28453,"num_write":353,"num_write_kb":45312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65142784,"data_stored":70402048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":4468736,"data_stored":5226633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215114,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31076,"kb_used_data":4044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340764,"statfs":{"total":96636764160,"available":96604942336,"internally_reserved":0,"allocated":4141056,"data_stored":4833417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215115,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33316,"kb_used_data":6284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":6434816,"data_stored":7385225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":4480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":4587520,"data_stored":4309221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280524,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":5732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":5869568,"data_stored":5597404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":5940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":6082560,"data_stored":5810396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29452,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342388,"statfs":{"total":96636764160,"available":96606605312,"internally_reserved":0,"allocated":2478080,"data_stored":2205916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":4280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":4382720,"data_stored":4110556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30228,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341612,"statfs":{"total":96636764160,"available":96605810688,"internally_reserved":0,"allocated":3276800,"data_stored":3000559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":4424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":4530176,"data_stored":4258012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36276,"kb_used_data":5088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335564,"statfs":{"total":96636764160,"available":96599617536,"internally_reserved":0,"allocated":5210112,"data_stored":4946666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38240,"kb_used_data":7048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333600,"statfs":{"total":96636764160,"available":96597606400,"internally_reserved":0,"allocated":7217152,"data_stored":8151177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247818,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36500,"kb_used_data":5304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335340,"statfs":{"total":96636764160,"available":96599388160,"internally_reserved":0,"allocated":5431296,"data_stored":6337193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247818,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30120,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341720,"statfs":{"total":96636764160,"available":96605921280,"internally_reserved":0,"allocated":3162112,"data_stored":4102219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36560,"kb_used_data":5372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335280,"statfs":{"total":96636764160,"available":96599326720,"internally_reserved":0,"allocated":5500928,"data_stored":7047224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30708,"kb_used_data":3676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341132,"statfs":{"total":96636764160,"available":96605319168,"internally_reserved":0,"allocated":3764224,"data_stored":4665481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6717440,"data_stored":7905280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4022272,"data_stored":4022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4161536,"data_stored":4161536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":3424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5320704,"data_stored":5320704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4423680,"data_stored":5128192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5500928,"data_stored":5500928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3264512,"data_stored":4419584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5533696,"data_stored":5533696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1929216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3379200,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3833856,"data_stored":3833856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2723840,"data_stored":2723840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4399104,"data_stored":5562368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":3575808,"data_stored":3575808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":3981312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:32.580 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T04:01:32.581 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:01:32.581 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-16T04:01:32.581 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T04:01:32.581 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T04:01:32.581 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-09-16T04:01:32.582 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:32.582 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:32.594 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:01:32.664 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:32.662+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:32.714 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:32.714 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T04:01:32.714 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:32.719 INFO:tasks.rados.rados.0.smithi188.stdout:143: done (0 left) 2024-09-16T04:01:32.719 INFO:tasks.rados.rados.0.smithi188.stdout:144: snap_create 2024-09-16T04:01:32.757 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:32.757 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:32.765 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:32.762+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:32.770 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:01:32.866 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:32.862+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:32.893 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:32.893 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:32.908 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:01:32.967 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:32.960+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:33.028 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:33.028 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:33.043 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:01:33.090 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:33.086+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:33.168 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:33.166+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:33.179 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:33.179 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:33.194 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:01:33.286 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:33.282+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:33.366 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:33.366 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:33.370 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:33.366+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:33.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:01:33.471 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:33.466+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:33.551 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:33.551 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:33.565 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:01:33.610 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:33.606+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:33.673 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:33.670+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:33.728 INFO:tasks.rados.rados.0.smithi188.stdout:144: done (0 left) 2024-09-16T04:01:33.728 INFO:tasks.rados.rados.0.smithi188.stdout:145: snap_remove snap 4 2024-09-16T04:01:33.740 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:01:33.740 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:33.741 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:33.755 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:01:33.773 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:33.770+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:33.875 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:33.874+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:33.878 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:33.878 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:33.893 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:01:33.975 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:33.974+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:34.018 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:34.018 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:34.032 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:01:34.076 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:34.074+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:34.161 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:34.161 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:34.177 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:01:34.178 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:34.172+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:34.278 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:34.278+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:34.304 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:34.304 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:34.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:01:34.371 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:01:34.417 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:34.412+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:34.441 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:34.441 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:34.454 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:01:34.502 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:34.498+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:34.531 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:34.531 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:34.549 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:34.549 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:34.570 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:34.570 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:34.581 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:34.578+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:34.581 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:34.582 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:34.591 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:34.591 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:34.596 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:01:34.613 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:34.613 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:34.634 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:34.635 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:34.657 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:34.658 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:34.681 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:34.689 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:34.686+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:34.703 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:34.703 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:34.726 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:34.726 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:34.728 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-09-16T04:01:34.734 INFO:tasks.rados.rados.0.smithi188.stdout:145: done (0 left) 2024-09-16T04:01:34.735 INFO:tasks.rados.rados.0.smithi188.stdout:146: snap_create 2024-09-16T04:01:34.752 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:34.753 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:34.775 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:34.776 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:34.777 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:34.778 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:34.782 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:34.782+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:34.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:01:34.795 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:34.795 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:34.815 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:34.816 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:34.839 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:34.839 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:34.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:34.836+0000 7fba164c1640 0 -- 172.21.15.37:0/318987871 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7fba10226340 msgr2=0x7fba10228730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:34.867 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:34.867 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:34.883 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:34.876+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.501908538000002, 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.56051925899999999, 2024-09-16T04:01:34.968 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460473+0000", 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.097e-06 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460488+0000", 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5335000000000001e-05 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460512+0000", 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4108999999999999e-05 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460533+0000", 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0721999999999999e-05 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460655+0000", 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122328 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460689+0000", 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014402+0000", 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.55371241699999996 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014725+0000", 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000323345 2024-09-16T04:01:34.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.015144+0000", 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041881500000000001 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020086+0000", 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049418869999999998 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020836+0000", 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074984300000000004 2024-09-16T04:01:34.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020926+0000", 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0070999999999999e-05 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020990+0000", 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4256000000000003e-05 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412334+0000", 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.550045877000002, 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015286879999999999, 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 113 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412334+0000", 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412334+0000", 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412334+0000", 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.23e-07 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412341+0000", 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1199999999999996e-06 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412354+0000", 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2935999999999999e-05 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412364+0000", 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7459999999999995e-06 2024-09-16T04:01:34.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.421535+0000", 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0091712579999999998 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.421664+0000", 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012879100000000001 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.421856+0000", 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019163399999999999 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425449+0000", 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035928940000000001 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427478+0000", 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0020294750000000002 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427558+0000", 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9526999999999999e-05 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427620+0000", 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2675999999999994e-05 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:34.980 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.550001442999999, 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016442578999999999, 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 114 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412379+0000", 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.06e-07 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412387+0000", 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1799999999999996e-06 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412397+0000", 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0236e-05 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412404+0000", 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2989999999999999e-06 2024-09-16T04:01:34.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.422787+0000", 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010382561 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.422907+0000", 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120316 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.423117+0000", 2024-09-16T04:01:34.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020991200000000001 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427100+0000", 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039832779999999998 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428687+0000", 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00158661 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428763+0000", 2024-09-16T04:01:34.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5679000000000005e-05 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428821+0000", 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7802e-05 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.986 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.549940446999997, 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01663504, 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 115 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:34.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412440+0000", 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.083e-06 2024-09-16T04:01:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412538+0000", 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7979999999999994e-05 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412552+0000", 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3736e-05 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412560+0000", 2024-09-16T04:01:34.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2630000000000003e-06 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424204+0000", 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011644046 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424328+0000", 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124282 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424537+0000", 2024-09-16T04:01:34.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020823900000000001 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427111+0000", 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025745849999999999 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428934+0000", 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018232680000000001 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429015+0000", 2024-09-16T04:01:34.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1034999999999999e-05 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429074+0000", 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8523000000000002e-05 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.549687067999997, 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016628581, 2024-09-16T04:01:34.992 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 116 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412693+0000", 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.077e-06 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412701+0000", 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8390000000000007e-06 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412715+0000", 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3482999999999999e-05 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412726+0000", 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0798999999999999e-05 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425538+0000", 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012812235999999999 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425665+0000", 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000126995 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425861+0000", 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019642699999999999 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427642+0000", 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001781244 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429160+0000", 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001517795 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429264+0000", 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010388299999999999 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429321+0000", 2024-09-16T04:01:34.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6802999999999998e-05 2024-09-16T04:01:34.998 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.549639560999999, 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019211259000000001, 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:34.999 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 117 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412741+0000", 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8700000000000005e-07 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412747+0000", 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1929999999999998e-06 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412757+0000", 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0039e-05 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412764+0000", 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3259999999999998e-06 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426386+0000", 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013621668999999999 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426547+0000", 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000161192 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426753+0000", 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020624599999999999 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428173+0000", 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001419305 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431799+0000", 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036262320000000001 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431890+0000", 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0987000000000001e-05 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431951+0000", 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1383000000000005e-05 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.004 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.196813837999997, 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016729104000000002, 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.005 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 160 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765567+0000", 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765573+0000", 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0260000000000004e-06 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765583+0000", 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765590+0000", 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4800000000000004e-06 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774469+0000", 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.008879184 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774666+0000", 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019695699999999999 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774998+0000", 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00033151099999999999 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778132+0000", 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031338849999999999 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782066+0000", 2024-09-16T04:01:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039342630000000003 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782181+0000", 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000115093 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782295+0000", 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113726 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.196777093999998, 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017211085000000001, 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 161 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765603+0000", 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:01:35.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765727+0000", 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124184 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765750+0000", 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2296000000000001e-05 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765759+0000", 2024-09-16T04:01:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2760000000000006e-06 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775766+0000", 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010007085000000001 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775856+0000", 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9683000000000002e-05 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.776003+0000", 2024-09-16T04:01:35.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014695999999999999 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779325+0000", 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033221930000000002 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782596+0000", 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032713519999999999 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782726+0000", 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013011500000000001 2024-09-16T04:01:35.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782813+0000", 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7174000000000003e-05 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.196602691999999, 2024-09-16T04:01:35.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017217072999999999, 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 162 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765778+0000", 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765784+0000", 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2659999999999998e-06 2024-09-16T04:01:35.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765794+0000", 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.029e-05 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765809+0000", 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4474999999999999e-05 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777438+0000", 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011628929 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777529+0000", 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1577999999999996e-05 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777666+0000", 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013660699999999999 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781937+0000", 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042713660000000004 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782885+0000", 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00094829699999999996 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782947+0000", 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1261999999999997e-05 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782994+0000", 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7139999999999999e-05 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.196552308999998, 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01977027, 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 163 2024-09-16T04:01:35.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765828+0000", 2024-09-16T04:01:35.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765834+0000", 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7830000000000004e-06 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765844+0000", 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0149000000000001e-05 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765852+0000", 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0199999999999994e-06 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778628+0000", 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012776183999999999 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778715+0000", 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6941000000000007e-05 2024-09-16T04:01:35.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778850+0000", 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013517800000000001 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783007+0000", 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041571960000000002 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785458+0000", 2024-09-16T04:01:35.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002450626 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785539+0000", 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1213999999999999e-05 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785597+0000", 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8258999999999999e-05 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.196506632000002, 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019887134000000001, 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 164 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.028 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765874+0000", 2024-09-16T04:01:35.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0130000000000001e-06 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765996+0000", 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122039 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766015+0000", 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9405000000000001e-05 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766025+0000", 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0096e-05 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779427+0000", 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013401569 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779649+0000", 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022229500000000001 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.780098+0000", 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00044856699999999998 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783018+0000", 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002919792 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785656+0000", 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026383489999999999 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785714+0000", 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8177000000000002e-05 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785760+0000", 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5831999999999998e-05 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.033 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.19633571, 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021760347999999999, 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 165 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766045+0000", 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766051+0000", 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8140000000000003e-06 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766062+0000", 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0592e-05 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766069+0000", 2024-09-16T04:01:35.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3629999999999996e-06 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781249+0000", 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015179349999999999 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781356+0000", 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000107747 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781496+0000", 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013985500000000001 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783295+0000", 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017986199999999999 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787679+0000", 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0043837559999999999 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787752+0000", 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3192e-05 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787804+0000", 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2182999999999998e-05 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.039 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.934710+0000", 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.027669344000003, 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013587647, 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 185 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934710+0000", 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934710+0000", 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934712+0000", 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.697e-06 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934903+0000", 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000191447 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934942+0000", 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8590999999999997e-05 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.934962+0000", 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0324999999999999e-05 2024-09-16T04:01:35.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.943925+0000", 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0089627860000000004 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.944057+0000", 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000132287 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.944272+0000", 2024-09-16T04:01:35.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021434199999999999 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948036+0000", 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037647499999999999 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948107+0000", 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0884999999999996e-05 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948187+0000", 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9961000000000002e-05 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948298+0000", 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011057600000000001 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.045 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935003+0000", 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.027376609999997, 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015768138000000001, 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 186 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935003+0000", 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935003+0000", 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935005+0000", 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7099999999999999e-06 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935017+0000", 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2836e-05 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935044+0000", 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6358000000000001e-05 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935062+0000", 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8515000000000001e-05 2024-09-16T04:01:35.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.945879+0000", 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010816345 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.945998+0000", 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000119162 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946187+0000", 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000189261 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949633+0000", 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00344641 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950612+0000", 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00097849500000000002 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950707+0000", 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4846999999999995e-05 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950771+0000", 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4199e-05 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.051 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.027290112999999, 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015994999999999999, 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 187 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935091+0000", 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.567e-06 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935103+0000", 2024-09-16T04:01:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1851999999999999e-05 2024-09-16T04:01:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935127+0000", 2024-09-16T04:01:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4596e-05 2024-09-16T04:01:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935142+0000", 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4986e-05 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946804+0000", 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011661985 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946926+0000", 2024-09-16T04:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000121455 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.947115+0000", 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018956100000000001 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950511+0000", 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033955729999999998 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950887+0000", 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037619500000000001 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951025+0000", 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013776399999999999 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951084+0000", 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9466000000000003e-05 2024-09-16T04:01:35.059 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.027212581000001, 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016322997999999998, 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.060 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 188 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935168+0000", 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4270000000000001e-06 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935319+0000", 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000150383 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935346+0000", 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7750999999999999e-05 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935392+0000", 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5361000000000002e-05 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948392+0000", 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013000429000000001 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948504+0000", 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011150900000000001 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948716+0000", 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021198300000000001 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951106+0000", 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0023904899999999999 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951365+0000", 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025906300000000002 2024-09-16T04:01:35.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951438+0000", 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2571999999999996e-05 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951490+0000", 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2030000000000002e-05 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.026952968000003, 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016713338000000001, 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 189 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.066 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935428+0000", 2024-09-16T04:01:35.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935443+0000", 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4599e-05 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935468+0000", 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4788000000000001e-05 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935486+0000", 2024-09-16T04:01:35.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8249000000000001e-05 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949701+0000", 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014214839 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949818+0000", 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011776499999999999 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950001+0000", 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018260100000000001 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951914+0000", 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001913049 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952007+0000", 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3006999999999999e-05 2024-09-16T04:01:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952083+0000", 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5722000000000004e-05 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952140+0000", 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7006000000000003e-05 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.071 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:346 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.922867+0000", 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.03951233, 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016365271000000001, 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 346 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922867+0000", 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922867+0000", 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922870+0000", 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9059999999999998e-06 2024-09-16T04:01:35.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922886+0000", 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5603e-05 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922921+0000", 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5583000000000001e-05 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922940+0000", 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8885999999999998e-05 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923091+0000", 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015139099999999999 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923104+0000", 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2924e-05 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.926969+0000", 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003865074 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927220+0000", 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025061199999999999 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.939232+0000", 2024-09-16T04:01:35.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012012292000000001 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:347 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.0393984600000001, 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018664877, 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.077 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 347 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.078 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.079 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922982+0000", 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.457e-06 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922991+0000", 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923032+0000", 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0878999999999999e-05 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923051+0000", 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9406e-05 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923225+0000", 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000173781 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923244+0000", 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8583000000000001e-05 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928577+0000", 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053332600000000003 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928836+0000", 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025873200000000002 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.941646+0000", 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01281 2024-09-16T04:01:35.082 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:348 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.0392652809999996, 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020600073999999999, 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:35.083 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 348 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923116+0000", 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8530000000000001e-06 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923127+0000", 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1396000000000001e-05 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923152+0000", 2024-09-16T04:01:35.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4253e-05 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923168+0000", 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6685999999999999e-05 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923332+0000", 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016361899999999999 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923353+0000", 2024-09-16T04:01:35.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0894999999999999e-05 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929889+0000", 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0065358930000000001 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.930145+0000", 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000256452 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:35.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:35.088 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.943714+0000", 2024-09-16T04:01:35.088 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013569027000000001 2024-09-16T04:01:35.088 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.088 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.088 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.088 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:35.088 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:35.088 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:35.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:01:35.090 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:35.086+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:35.191 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:35.186+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:35.223 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:35.223 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:35.223 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:35.223 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:35.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:01:35.291 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:35.286+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:35.360 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:35.360 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:35.360 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:35.361 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:35.361 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:35.374 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:01:35.392 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:35.390+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:35.493 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:35.488+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:35.542 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:35.542 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:35.542 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:35.542 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:01:35.542 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.542 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:54 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'32, mlcod=25'29)", 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 36.409759936999997, 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005928254, 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.543 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128770+0000", 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128784+0000", 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3959e-05 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.544 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128819+0000", 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5237999999999997e-05 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128838+0000", 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8754999999999999e-05 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128996+0000", 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000158419 2024-09-16T04:01:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129033+0000", 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6905e-05 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.134608+0000", 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055745869999999998 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.134697+0000", 2024-09-16T04:01:35.546 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8701000000000003e-05 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:55 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'33, mlcod=25'29)", 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.129935+0000", 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 36.408593586999999, 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005436906, 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.547 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129935+0000", 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129935+0000", 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.548 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129936+0000", 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.556e-06 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130009+0000", 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.2582000000000004e-05 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130033+0000", 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4032e-05 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.549 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130046+0000", 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2699e-05 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130161+0000", 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00011516899999999999 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.130194+0000", 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2795e-05 2024-09-16T04:01:35.550 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.135305+0000", 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051117280000000003 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.135372+0000", 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6345000000000003e-05 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.551 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:87 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'57, mlcod=25'51)", 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 36.340276226999997, 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056693940000000003, 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.552 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198253+0000", 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2699999999999999e-06 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198371+0000", 2024-09-16T04:01:35.553 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000117349 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198394+0000", 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3768e-05 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198411+0000", 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6186000000000001e-05 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.554 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198538+0000", 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000127842 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198574+0000", 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5451000000000003e-05 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.203863+0000", 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052887919999999996 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.555 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.203921+0000", 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8736000000000001e-05 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:120 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'82, mlcod=25'78)", 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.435845+0000", 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 36.102683278000001, 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052667130000000001, 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.556 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435845+0000", 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435845+0000", 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.557 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435847+0000", 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6530000000000001e-06 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435928+0000", 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1687000000000006e-05 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435957+0000", 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8694999999999999e-05 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.558 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.435970+0000", 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3315999999999999e-05 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436070+0000", 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.9270000000000003e-05 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436106+0000", 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6021e-05 2024-09-16T04:01:35.559 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.441056+0000", 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0049504750000000002 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.441112+0000", 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5596000000000001e-05 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.560 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:121 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'83, mlcod=25'78)", 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 36.101636630999998, 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0070855379999999997, 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.561 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436894+0000", 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8700000000000001e-06 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436963+0000", 2024-09-16T04:01:35.562 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9152000000000005e-05 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436991+0000", 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8388e-05 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.437006+0000", 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4883e-05 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.563 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.438211+0000", 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0012053599999999999 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.438267+0000", 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5810000000000003e-05 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.443897+0000", 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056298700000000004 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.564 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.443977+0000", 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0204999999999995e-05 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:129 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'87, mlcod=25'85)", 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.945283519, 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054743200000000004, 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.565 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593246+0000", 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6759999999999999e-06 2024-09-16T04:01:35.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593258+0000", 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1715999999999999e-05 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593286+0000", 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7912e-05 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593299+0000", 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3352e-05 2024-09-16T04:01:35.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593426+0000", 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000127002 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593482+0000", 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5952999999999998e-05 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.568 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598617+0000", 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051345549999999998 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598719+0000", 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000102154 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:130 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'88, mlcod=25'85)", 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.593454+0000", 2024-09-16T04:01:35.569 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.945074265999999, 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053433049999999996, 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593454+0000", 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593454+0000", 2024-09-16T04:01:35.570 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593456+0000", 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6130000000000001e-06 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593466+0000", 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0686000000000001e-05 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.571 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593486+0000", 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9542e-05 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593500+0000", 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3776e-05 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594031+0000", 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00053142100000000002 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.572 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594069+0000", 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8244000000000002e-05 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598727+0000", 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0046581239999999996 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598797+0000", 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9899000000000003e-05 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.573 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:131 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'89, mlcod=25'85)", 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.943860933000003, 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058446720000000004, 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.574 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594669+0000", 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0700000000000001e-06 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.575 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594744+0000", 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4637999999999999e-05 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594769+0000", 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5245e-05 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594782+0000", 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2840000000000001e-05 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.576 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594871+0000", 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.9091000000000005e-05 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594901+0000", 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9751e-05 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600445+0000", 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055442649999999996 2024-09-16T04:01:35.577 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600512+0000", 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6772000000000004e-05 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:132 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'90, mlcod=25'85)", 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.943714069000002, 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005744519, 2024-09-16T04:01:35.578 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.579 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594815+0000", 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2900000000000002e-07 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594825+0000", 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5640000000000002e-06 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594836+0000", 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1025e-05 2024-09-16T04:01:35.580 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594844+0000", 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8699999999999998e-06 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595492+0000", 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00064781699999999997 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595549+0000", 2024-09-16T04:01:35.581 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6842999999999998e-05 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600518+0000", 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004968386 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600559+0000", 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1185e-05 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.582 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:133 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'91, mlcod=25'85)", 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.943660313000002, 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057530580000000001, 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.37e-07 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.584 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594874+0000", 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3890000000000004e-06 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594884+0000", 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8970000000000003e-06 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594890+0000", 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.5989999999999998e-06 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.585 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596024+0000", 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0011332289999999999 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596076+0000", 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2253000000000003e-05 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600563+0000", 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004487301 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.586 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600621+0000", 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7853000000000003e-05 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:134 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'92, mlcod=25'85)", 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.943178074999999, 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0068484870000000003, 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.587 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.588 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595352+0000", 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595414+0000", 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1926000000000003e-05 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595439+0000", 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4794999999999999e-05 2024-09-16T04:01:35.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595453+0000", 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4283e-05 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596809+0000", 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001356407 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596847+0000", 2024-09-16T04:01:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7731000000000002e-05 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602111+0000", 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052641040000000004 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602199+0000", 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7551000000000003e-05 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:135 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'93, mlcod=25'85)", 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.942508504999999, 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006226517, 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596021+0000", 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.403e-06 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596033+0000", 2024-09-16T04:01:35.593 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1888999999999999e-05 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596051+0000", 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8091999999999999e-05 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596065+0000", 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4063000000000001e-05 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.594 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597461+0000", 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001395568 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597495+0000", 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3824e-05 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602202+0000", 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0047076369999999998 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.595 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602246+0000", 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4041000000000001e-05 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:136 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'94, mlcod=25'85)", 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.942006348, 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057697900000000003, 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.596 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596524+0000", 2024-09-16T04:01:35.597 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.796e-06 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596536+0000", 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1963e-05 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596560+0000", 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3855000000000001e-05 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.598 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596572+0000", 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2836e-05 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597942+0000", 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0013696769999999999 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598001+0000", 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8856e-05 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.599 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602248+0000", 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042472500000000002 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602292+0000", 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3557000000000003e-05 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.600 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:137 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'95, mlcod=25'85)", 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.596971+0000", 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.941557240999998, 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053633309999999998, 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596971+0000", 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.601 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596971+0000", 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596972+0000", 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1999999999999999e-06 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597033+0000", 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.0921999999999997e-05 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.602 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597051+0000", 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7448999999999998e-05 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597061+0000", 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.007e-05 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598891+0000", 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0018303589999999999 2024-09-16T04:01:35.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598927+0000", 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6168000000000003e-05 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602297+0000", 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003369698 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.604 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602334+0000", 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7465000000000001e-05 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:163 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'117, mlcod=25'112)", 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.759622759000003, 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061538799999999996, 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.605 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778907+0000", 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.08e-06 2024-09-16T04:01:35.606 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778932+0000", 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5661999999999999e-05 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778950+0000", 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7852000000000001e-05 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.607 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778961+0000", 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1333e-05 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.779058+0000", 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6280000000000007e-05 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.779087+0000", 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9505e-05 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.608 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.784992+0000", 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059048049999999999 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.785059+0000", 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7362999999999999e-05 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:165 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'119, mlcod=25'112)", 2024-09-16T04:01:35.609 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.756919269999997, 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056794660000000002, 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:35.610 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781611+0000", 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.827e-06 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781629+0000", 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7734999999999999e-05 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.611 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781657+0000", 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8065e-05 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781670+0000", 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2993000000000001e-05 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781828+0000", 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000158436 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.612 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781846+0000", 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7651000000000001e-05 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.787255+0000", 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054093639999999998 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.787288+0000", 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3395000000000001e-05 2024-09-16T04:01:35.613 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s1 40/33 ECSubWrite(tid=191, reqid=client.4646.0:318, at_version=40'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.027929+0000", 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.5105988680000006, 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0060192860000000004, 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027929+0000", 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027929+0000", 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027930+0000", 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7000000000000004e-07 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028366+0000", 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00043583899999999999 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028399+0000", 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2872e-05 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028416+0000", 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6889e-05 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028558+0000", 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014243599999999999 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028574+0000", 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5605999999999999e-05 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033846+0000", 2024-09-16T04:01:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052723049999999997 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033949+0000", 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000102569 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s1 40/33 ECSubWrite(tid=193, reqid=client.4646.0:320, at_version=40'15, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.032369+0000", 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.5061594700000001, 2024-09-16T04:01:35.618 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051532039999999998, 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.032369+0000", 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.032369+0000", 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.619 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.032371+0000", 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1730000000000002e-06 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.032519+0000", 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014823899999999999 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.032557+0000", 2024-09-16T04:01:35.620 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7568999999999999e-05 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.032580+0000", 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2759e-05 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.032752+0000", 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017273500000000001 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:35.621 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.032776+0000", 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3881999999999999e-05 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.037421+0000", 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0046448110000000004 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.037522+0000", 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000101036 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 40/33 ECSubWrite(tid=51, reqid=client.4646.0:323, at_version=40'28, trim_to=0'0, roll_forward_to=40'26))", 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.033639+0000", 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.5048889750000001, 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0063318030000000004, 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033639+0000", 2024-09-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033639+0000", 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033641+0000", 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.35e-06 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033692+0000", 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.1409000000000002e-05 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033730+0000", 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8229000000000003e-05 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033752+0000", 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1705e-05 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.625 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033936+0000", 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000183604 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033960+0000", 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4253e-05 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039854+0000", 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058937670000000003 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.626 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039971+0000", 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00011748600000000001 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s1 40/33 ECSubWrite(tid=195, reqid=client.4646.0:321, at_version=0'0, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.039640+0000", 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.4988878280000009, 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054764610000000002, 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:35.627 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039640+0000", 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039640+0000", 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:35.628 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039642+0000", 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.903e-06 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039670+0000", 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7279000000000001e-05 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039703+0000", 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3108999999999998e-05 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039726+0000", 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3238999999999999e-05 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039952+0000", 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00022652999999999999 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.040026+0000", 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3566999999999995e-05 2024-09-16T04:01:35.630 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.045041+0000", 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0050146260000000003 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.045117+0000", 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6207999999999999e-05 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:35.631 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:35.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:01:35.634 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:35.630+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:35.734 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:35.728+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:35.746 INFO:tasks.rados.rados.0.smithi188.stdout:146: done (0 left) 2024-09-16T04:01:35.746 INFO:tasks.rados.rados.0.smithi188.stdout:147: read oid 29 snap 8 2024-09-16T04:01:35.746 INFO:tasks.rados.rados.0.smithi188.stdout:147: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T04:01:35.747 INFO:tasks.rados.rados.0.smithi188.stdout:148: rollback oid 39 current snap is 10 2024-09-16T04:01:35.747 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 39 to 3 2024-09-16T04:01:35.747 INFO:tasks.rados.rados.0.smithi188.stdout:149: snap_remove snap 9 2024-09-16T04:01:35.762 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:35.763 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:35.763 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:35.763 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:35.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:01:35.835 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:35.834+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:35.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:01:35.936 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:35.930+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:36.037 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:36.028+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s3 40/33 ECSubWrite(tid=191, reqid=client.4646.0:318, at_version=40'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.027913+0000", 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.010630515000001, 2024-09-16T04:01:36.042 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006972392, 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027913+0000", 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027913+0000", 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027914+0000", 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7399999999999999e-07 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028231+0000", 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000316869 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028265+0000", 2024-09-16T04:01:36.044 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4705000000000001e-05 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028291+0000", 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5182999999999999e-05 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028430+0000", 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00013959199999999999 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.045 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028456+0000", 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6055999999999999e-05 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.034798+0000", 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0063422330000000001 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.034886+0000", 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7379999999999993e-05 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.046 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s3 40/33 ECSubWrite(tid=195, reqid=client.4646.0:321, at_version=0'0, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.039728+0000", 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.9988156180000001, 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057121029999999996, 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039728+0000", 2024-09-16T04:01:36.047 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039728+0000", 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039730+0000", 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0059999999999999e-06 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.048 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039759+0000", 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8606e-05 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039807+0000", 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8097999999999999e-05 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039829+0000", 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2113e-05 2024-09-16T04:01:36.049 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039980+0000", 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015066499999999999 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.040005+0000", 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4879e-05 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.045347+0000", 2024-09-16T04:01:36.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053424620000000001 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.045440+0000", 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.3274000000000002e-05 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=55, reqid=client.4646.0:327, at_version=41'32, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.895757+0000", 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1427873739999992, 2024-09-16T04:01:36.051 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0075359370000000004, 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895757+0000", 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.052 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895757+0000", 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895757+0000", 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895828+0000", 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.0853999999999996e-05 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.053 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895862+0000", 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3896000000000002e-05 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895887+0000", 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4732999999999999e-05 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.896051+0000", 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016381699999999999 2024-09-16T04:01:36.054 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.896077+0000", 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6075999999999999e-05 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903206+0000", 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0071296729999999996 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903292+0000", 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6098000000000001e-05 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=56, reqid=client.4646.0:328, at_version=41'33, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.898528+0000", 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1400155180000002, 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0091371349999999994, 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.056 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898528+0000", 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898528+0000", 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.057 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898530+0000", 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898581+0000", 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0807999999999999e-05 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898624+0000", 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3244999999999997e-05 2024-09-16T04:01:36.058 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898651+0000", 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6716e-05 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898817+0000", 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016536099999999999 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.059 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898841+0000", 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4796000000000001e-05 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907533+0000", 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0086919360000000008 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907666+0000", 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000132309 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.060 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.bs4 41/33 ECSubWrite(tid=69, reqid=osd.4.0:69, at_version=41'20, trim_to=0'0, roll_forward_to=37'18))", 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1399542599999997, 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079802759999999997, 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.061 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.4700000000000001e-07 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.062 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898652+0000", 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1468000000000002e-05 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898683+0000", 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1025999999999997e-05 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898705+0000", 2024-09-16T04:01:36.063 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2436e-05 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898898+0000", 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019302300000000001 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898922+0000", 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4202999999999998e-05 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906462+0000", 2024-09-16T04:01:36.064 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0075398959999999999 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906570+0000", 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000107577 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/33 ECSubWrite(tid=10, reqid=osd.5.0:10, at_version=41'8, trim_to=0'0, roll_forward_to=37'6))", 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.901860+0000", 2024-09-16T04:01:36.065 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1366840029999992, 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0097700600000000005, 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901860+0000", 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901860+0000", 2024-09-16T04:01:36.066 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901861+0000", 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901920+0000", 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.9755e-05 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.067 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901951+0000", 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0289000000000001e-05 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901975+0000", 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3929000000000001e-05 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902135+0000", 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016060099999999999 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.068 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902159+0000", 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4139999999999999e-05 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911537+0000", 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093778990000000003 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911630+0000", 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2783999999999998e-05 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.069 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=57, reqid=client.4646.0:329, at_version=41'34, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.902328+0000", 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1362162659999999, 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093967600000000005, 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.070 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902328+0000", 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902328+0000", 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902330+0000", 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7439999999999999e-06 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.071 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902376+0000", 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.5858e-05 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902412+0000", 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5281999999999999e-05 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902436+0000", 2024-09-16T04:01:36.072 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4326000000000001e-05 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902591+0000", 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015515200000000001 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902621+0000", 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9816e-05 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.073 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911638+0000", 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0090172280000000004 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911724+0000", 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6353999999999993e-05 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.bs4 41/33 ECSubWrite(tid=71, reqid=osd.4.0:69, at_version=0'0, trim_to=0'0, roll_forward_to=41'20))", 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.907338+0000", 2024-09-16T04:01:36.074 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1312062820000008, 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0070074170000000002, 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907338+0000", 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907338+0000", 2024-09-16T04:01:36.075 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907346+0000", 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0520000000000002e-06 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907371+0000", 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5720000000000001e-05 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.076 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907399+0000", 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8019000000000001e-05 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907421+0000", 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1455999999999999e-05 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907589+0000", 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016838399999999999 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.077 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907616+0000", 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6582999999999999e-05 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914241+0000", 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0066255159999999997 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914345+0000", 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000103687 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.078 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=58, reqid=client.4646.0:330, at_version=41'35, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.907383+0000", 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1311613509999994, 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0087796330000000002, 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.079 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907383+0000", 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907383+0000", 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907385+0000", 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0439999999999998e-06 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.080 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907467+0000", 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2282999999999998e-05 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907508+0000", 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1242999999999998e-05 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907535+0000", 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7078999999999999e-05 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.081 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908206+0000", 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00067093399999999998 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908229+0000", 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2875999999999999e-05 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916072+0000", 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0078425939999999996 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.082 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916162+0000", 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.0580000000000004e-05 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 41/33 ECSubWrite(tid=23, reqid=osd.15.0:23, at_version=41'17, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.910049+0000", 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1284951779999997, 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058227060000000004, 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.083 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910049+0000", 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910049+0000", 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.084 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910050+0000", 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910103+0000", 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3260999999999997e-05 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910133+0000", 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0533000000000001e-05 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.085 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910155+0000", 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2178999999999999e-05 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910326+0000", 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000170168 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910349+0000", 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3422e-05 2024-09-16T04:01:36.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915748+0000", 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053994100000000003 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915871+0000", 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012296599999999999 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.087 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 41/33 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=41'19, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.910207+0000", 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1283370769999994, 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079503769999999998, 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910207+0000", 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910207+0000", 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910208+0000", 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0440000000000001e-06 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910261+0000", 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2645000000000002e-05 2024-09-16T04:01:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910283+0000", 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2841999999999999e-05 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910301+0000", 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8076000000000001e-05 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911107+0000", 2024-09-16T04:01:36.090 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00080574399999999995 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911130+0000", 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2685999999999999e-05 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918072+0000", 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0069422989999999999 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.091 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918157+0000", 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.5041000000000001e-05 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=59, reqid=osd.14.0:59, at_version=41'37, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.912180+0000", 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1263639780000005, 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0060600319999999999, 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.092 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.912180+0000", 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.912180+0000", 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.093 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.912181+0000", 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.44e-06 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.912230+0000", 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8331999999999997e-05 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.912263+0000", 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3139000000000002e-05 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.094 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.912284+0000", 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0959e-05 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.912429+0000", 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000144766 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.912452+0000", 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3609e-05 2024-09-16T04:01:36.095 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918151+0000", 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005698407 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918240+0000", 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.9380000000000001e-05 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.096 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=60, reqid=osd.14.0:60, at_version=41'39, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.914069+0000", 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1244745100000006, 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010458115, 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914069+0000", 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.097 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914069+0000", 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914071+0000", 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9700000000000002e-06 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914182+0000", 2024-09-16T04:01:36.098 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00011102 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914216+0000", 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3915999999999999e-05 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914240+0000", 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3768999999999999e-05 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.099 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914400+0000", 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015951100000000001 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914425+0000", 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4922999999999998e-05 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924429+0000", 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010004341999999999 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.100 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924528+0000", 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8664000000000003e-05 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 41/33 ECSubWrite(tid=101, reqid=osd.11.0:101, at_version=41'54, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1221736480000004, 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011980414999999999, 2024-09-16T04:01:36.101 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.102 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916371+0000", 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6300000000000004e-07 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916424+0000", 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2497999999999998e-05 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916451+0000", 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7673000000000001e-05 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.103 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916473+0000", 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1573000000000002e-05 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916863+0000", 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00039059000000000001 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916888+0000", 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4629000000000001e-05 2024-09-16T04:01:36.104 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928269+0000", 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011381092000000001 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928351+0000", 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1596999999999998e-05 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.105 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1220153929999999, 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015945876000000001, 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.106 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916530+0000", 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.133e-06 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916571+0000", 2024-09-16T04:01:36.107 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1539000000000003e-05 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916594+0000", 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2915999999999999e-05 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916613+0000", 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8766e-05 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.108 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918993+0000", 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0023803050000000001 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919007+0000", 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3703000000000001e-05 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932390+0000", 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013382897 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.109 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932474+0000", 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.4617000000000007e-05 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.as5 41/33 ECSubWrite(tid=13, reqid=osd.3.0:13, at_version=41'8, trim_to=0'0, roll_forward_to=38'6))", 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1212916100000001, 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010948411999999999, 2024-09-16T04:01:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.111 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917253+0000", 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7299999999999996e-07 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917309+0000", 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5988e-05 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.112 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917340+0000", 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1421999999999998e-05 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917364+0000", 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3893e-05 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917525+0000", 2024-09-16T04:01:36.113 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016099500000000001 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917551+0000", 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5344999999999999e-05 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928110+0000", 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010559192 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.114 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928201+0000", 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.1003999999999998e-05 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/33 ECSubWrite(tid=197, reqid=osd.13.0:197, at_version=41'18, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1194735199999997, 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013473318999999999, 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.115 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.116 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919071+0000", 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7e-07 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919120+0000", 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.9595000000000002e-05 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919147+0000", 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6415999999999999e-05 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.117 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919176+0000", 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9002000000000001e-05 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919332+0000", 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015599200000000001 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919355+0000", 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3312e-05 2024-09-16T04:01:36.118 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932433+0000", 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013077961000000001 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932544+0000", 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000110571 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.119 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/33 ECSubWrite(tid=199, reqid=osd.13.0:199, at_version=41'20, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1177125270000001, 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013627653999999999, 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.120 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920833+0000", 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.809e-06 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.121 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920890+0000", 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6799000000000003e-05 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920921+0000", 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1161999999999997e-05 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920944+0000", 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3198999999999999e-05 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.122 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.926110+0000", 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051653740000000004 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.926136+0000", 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6568999999999999e-05 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.934366+0000", 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0082292779999999996 2024-09-16T04:01:36.123 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.934459+0000", 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.3463999999999998e-05 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 41/33 ECSubWrite(tid=25, reqid=osd.15.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=41'19))", 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.923453+0000", 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1150913500000001, 2024-09-16T04:01:36.124 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093221420000000003, 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923453+0000", 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923453+0000", 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.125 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923455+0000", 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0829999999999998e-06 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923481+0000", 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6463000000000001e-05 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923909+0000", 2024-09-16T04:01:36.126 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00042800899999999997 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923939+0000", 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9916e-05 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924156+0000", 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000216799 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.127 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924181+0000", 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4966000000000001e-05 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932674+0000", 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0084933649999999992 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932775+0000", 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000100541 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.128 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=61, reqid=osd.14.0:60, at_version=0'0, trim_to=0'0, roll_forward_to=41'39))", 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.928063+0000", 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.1104813060000005, 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006207855, 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.129 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928063+0000", 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928063+0000", 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928064+0000", 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.751e-06 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.130 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928090+0000", 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5616e-05 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928123+0000", 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2891000000000001e-05 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928146+0000", 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2702e-05 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.131 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928280+0000", 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000134838 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928305+0000", 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4428999999999999e-05 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.934165+0000", 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005859909 2024-09-16T04:01:36.132 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.934270+0000", 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000105719 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.133 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:36.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:01:36.138 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:36.134+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:36.238 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:36.234+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:36.264 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:36.264 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:36.264 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:36.264 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:36.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:01:36.339 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:36.338+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:36.441 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:36.438+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:36.448 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:36.448 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:36.448 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:36.448 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:36.448 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:36.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:01:36.541 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:36.536+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:36.583 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:36.583 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:36.583 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:36.583 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:01:36.583 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.583 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 11.690976535000001, 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062540319999999996, 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.584 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.13e-07 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889239+0000", 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000639432 2024-09-16T04:01:36.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889269+0000", 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0131999999999999e-05 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889293+0000", 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4396000000000002e-05 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889486+0000", 2024-09-16T04:01:36.586 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019278000000000001 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889535+0000", 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8442000000000001e-05 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894775+0000", 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052406800000000002 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894853+0000", 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7557000000000003e-05 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 39/33 ECSubWrite(tid=82, reqid=client.4646.0:293, at_version=39'43, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:24.894136+0000", 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 11.685438955, 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0031695130000000001, 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.588 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894136+0000", 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894136+0000", 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.589 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894138+0000", 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5629999999999999e-06 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894174+0000", 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6226000000000002e-05 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894196+0000", 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1962000000000001e-05 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.590 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894217+0000", 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0423000000000001e-05 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894360+0000", 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014322200000000001 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894382+0000", 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2382e-05 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.591 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.897222+0000", 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.002839528 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.897306+0000", 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.4207000000000002e-05 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.592 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 40/33 ECSubWrite(tid=85, reqid=client.4646.0:297, at_version=40'45, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:25.907031+0000", 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.672544384, 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0050522809999999996, 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907031+0000", 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.593 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907031+0000", 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907090+0000", 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8811000000000003e-05 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907534+0000", 2024-09-16T04:01:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000443759 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907571+0000", 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7509000000000003e-05 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907594+0000", 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3394999999999999e-05 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.595 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907748+0000", 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000153659 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907778+0000", 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0349e-05 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.912008+0000", 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042299080000000001 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.596 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.912083+0000", 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4890999999999998e-05 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 40/33 ECSubWrite(tid=87, reqid=client.4646.0:298, at_version=40'48, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:25.916888+0000", 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.662687500000001, 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0029958559999999999, 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.597 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.916888+0000", 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.916888+0000", 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.598 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.916889+0000", 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3799999999999999e-06 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.917230+0000", 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00034066899999999999 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.917256+0000", 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6485000000000001e-05 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.599 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.917272+0000", 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5466999999999999e-05 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.917437+0000", 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016531099999999999 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.917464+0000", 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6979e-05 2024-09-16T04:01:36.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.919578+0000", 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0021143580000000002 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.919884+0000", 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00030520699999999999 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.601 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 40/33 ECSubWrite(tid=66, reqid=client.4646.0:302, at_version=40'8, trim_to=0'0, roll_forward_to=37'6))", 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:25.919731+0000", 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.659844269000001, 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004058379, 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.919731+0000", 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.602 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.919731+0000", 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.919732+0000", 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.4300000000000003e-07 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.920164+0000", 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00043205899999999999 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.603 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.920203+0000", 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8958999999999997e-05 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.920227+0000", 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4482e-05 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.920398+0000", 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017125500000000001 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.604 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.920427+0000", 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8716000000000001e-05 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.923675+0000", 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0032473229999999999 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.923789+0000", 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000114942 2024-09-16T04:01:36.605 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 40/33 ECSubWrite(tid=88, reqid=client.4646.0:298, at_version=0'0, trim_to=0'0, roll_forward_to=40'48))", 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:25.921311+0000", 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.658264473999999, 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003029623, 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.606 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.921311+0000", 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.921311+0000", 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.921313+0000", 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0430000000000002e-06 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.607 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.921344+0000", 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0685000000000002e-05 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.921381+0000", 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7771999999999997e-05 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.921406+0000", 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4956e-05 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.608 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.921545+0000", 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00013819900000000001 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.921570+0000", 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5369000000000001e-05 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.924246+0000", 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0026765460000000001 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.609 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.924340+0000", 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.4053000000000002e-05 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 40/33 ECSubWrite(tid=95, reqid=client.4646.0:310, at_version=40'17, trim_to=0'0, roll_forward_to=37'14))", 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:25.996281+0000", 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.583294385, 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0033064790000000002, 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.610 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.996281+0000", 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.996281+0000", 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.611 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.996283+0000", 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5270000000000001e-06 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.996760+0000", 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00047657500000000002 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.996803+0000", 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3307999999999999e-05 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.612 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.996827+0000", 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3929000000000001e-05 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.996949+0000", 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000121403 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.996964+0000", 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5209e-05 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.613 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.999491+0000", 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0025267419999999998 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.999587+0000", 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6786000000000005e-05 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.614 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 40/33 ECSubWrite(tid=97, reqid=client.4646.0:312, at_version=40'19, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.001218+0000", 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.578357063, 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0041152319999999999, 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.001218+0000", 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.615 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.001218+0000", 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.001226+0000", 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7200000000000006e-06 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.001271+0000", 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.5334999999999999e-05 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.001304+0000", 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2499000000000002e-05 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.001331+0000", 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6855999999999998e-05 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.001593+0000", 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00026269200000000003 2024-09-16T04:01:36.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.001618+0000", 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4885e-05 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.005262+0000", 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0036440550000000002 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.005334+0000", 2024-09-16T04:01:36.618 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1190000000000001e-05 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/33 ECSubWrite(tid=68, reqid=osd.4.0:68, at_version=41'10, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.899589+0000", 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6799865500000006, 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0041137750000000001, 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.619 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899589+0000", 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899589+0000", 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899590+0000", 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4700000000000001e-07 2024-09-16T04:01:36.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899648+0000", 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8418000000000002e-05 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899672+0000", 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4532000000000002e-05 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899696+0000", 2024-09-16T04:01:36.621 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3070000000000001e-05 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899874+0000", 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017796700000000001 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899898+0000", 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4448999999999999e-05 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903584+0000", 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003685809 2024-09-16T04:01:36.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903703+0000", 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000118783 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/33 ECSubWrite(tid=70, reqid=osd.4.0:70, at_version=41'12, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6792551010000007, 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059225500000000004, 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.623 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900322+0000", 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.894e-06 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900375+0000", 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2994000000000001e-05 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900405+0000", 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9543000000000002e-05 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.625 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900447+0000", 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.2824999999999997e-05 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902403+0000", 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001955666 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902429+0000", 2024-09-16T04:01:36.626 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5783e-05 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906151+0000", 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037224810000000001 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906243+0000", 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.1364000000000001e-05 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.ds3 41/33 ECSubWrite(tid=28, reqid=osd.7.0:28, at_version=41'17, trim_to=0'0, roll_forward_to=38'15))", 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.906554+0000", 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6730215749999999, 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0048378750000000002, 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906554+0000", 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.628 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906554+0000", 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906554+0000", 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8400000000000004e-07 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906626+0000", 2024-09-16T04:01:36.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1054000000000001e-05 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906655+0000", 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9278999999999999e-05 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906678+0000", 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2986e-05 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.630 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906875+0000", 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019720999999999999 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906890+0000", 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4786e-05 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911303+0000", 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004413419 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911392+0000", 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8456999999999997e-05 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.ds3 41/33 ECSubWrite(tid=29, reqid=osd.7.0:29, at_version=41'19, trim_to=0'0, roll_forward_to=38'15))", 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6709407029999994, 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053361119999999996, 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.632 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.633 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908637+0000", 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9269999999999999e-06 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908688+0000", 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908719+0000", 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1479000000000001e-05 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.634 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908741+0000", 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2152999999999999e-05 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908934+0000", 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019235000000000001 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908957+0000", 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3572000000000001e-05 2024-09-16T04:01:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.913883+0000", 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0049257820000000001 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.913971+0000", 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7854000000000003e-05 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.636 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/33 ECSubWrite(tid=101, reqid=osd.11.0:101, at_version=41'54, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6655584949999991, 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062086830000000004, 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.637 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7999999999999995e-07 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914081+0000", 2024-09-16T04:01:36.638 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.3151000000000006e-05 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914110+0000", 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9672e-05 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914133+0000", 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2439e-05 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.639 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914712+0000", 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00057882500000000004 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914737+0000", 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920147+0000", 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054103859999999997 2024-09-16T04:01:36.640 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920226+0000", 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8573999999999997e-05 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 41/33 ECSubWrite(tid=100, reqid=osd.11.0:100, at_version=41'21, trim_to=0'0, roll_forward_to=40'19))", 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6653877309999991, 2024-09-16T04:01:36.641 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0078425089999999992, 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.642 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914189+0000", 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2999999999999999e-07 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914239+0000", 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0232000000000003e-05 2024-09-16T04:01:36.643 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914261+0000", 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2282000000000001e-05 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914279+0000", 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8283000000000001e-05 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.644 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915647+0000", 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0013673719999999999 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915669+0000", 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2198999999999999e-05 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921948+0000", 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062792710000000003 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.645 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922030+0000", 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1940000000000005e-05 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/33 ECSubWrite(tid=72, reqid=osd.4.0:70, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6645263030000006, 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0075338999999999996, 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.646 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.647 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915051+0000", 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.057e-06 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915081+0000", 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9488999999999998e-05 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915114+0000", 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3312000000000002e-05 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.648 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915138+0000", 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4450000000000001e-05 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915351+0000", 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00021263900000000001 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915375+0000", 2024-09-16T04:01:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4219999999999999e-05 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922495+0000", 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0071197559999999997 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922583+0000", 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7977000000000001e-05 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.650 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6626817230000004, 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0069081630000000002, 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.651 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916895+0000", 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.88e-06 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916949+0000", 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3551999999999998e-05 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.652 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916983+0000", 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4178999999999999e-05 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917008+0000", 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5049e-05 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917176+0000", 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016821800000000001 2024-09-16T04:01:36.653 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917201+0000", 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4695999999999999e-05 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923709+0000", 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0065079020000000003 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923802+0000", 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2687000000000002e-05 2024-09-16T04:01:36.654 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.ds3 41/33 ECSubWrite(tid=30, reqid=osd.7.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=41'19))", 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.920075+0000", 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6595006469999998, 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003676393, 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.655 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920075+0000", 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920075+0000", 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920077+0000", 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9130000000000001e-06 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.656 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920106+0000", 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9298999999999998e-05 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920138+0000", 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1986000000000002e-05 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920161+0000", 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3339000000000001e-05 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.657 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920383+0000", 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00022169299999999999 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920407+0000", 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4572000000000001e-05 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923665+0000", 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003257687 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.658 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923751+0000", 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.5903999999999996e-05 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 41/33 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=41'23, trim_to=0'0, roll_forward_to=40'19))", 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.924677+0000", 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6548979040000003, 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0041578969999999998, 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.659 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924677+0000", 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924677+0000", 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.660 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924680+0000", 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1390000000000002e-06 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924768+0000", 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8038e-05 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924811+0000", 2024-09-16T04:01:36.661 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3553999999999997e-05 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924835+0000", 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3632999999999999e-05 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924976+0000", 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000141509 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.662 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.925001+0000", 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4442000000000001e-05 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928747+0000", 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037464920000000001 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928835+0000", 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8090000000000005e-05 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/33 ECSubWrite(tid=106, reqid=osd.11.0:106, at_version=41'58, trim_to=0'0, roll_forward_to=41'56))", 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.936946+0000", 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6426297620000003, 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003432526, 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.936946+0000", 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.936946+0000", 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.936948+0000", 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6860000000000002e-06 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.936988+0000", 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.9845000000000002e-05 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937012+0000", 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3972999999999999e-05 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937031+0000", 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8955999999999999e-05 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937181+0000", 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014966400000000001 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937194+0000", 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3471e-05 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.940290+0000", 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0030958100000000001 2024-09-16T04:01:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.940378+0000", 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8121000000000006e-05 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/33 ECSubWrite(tid=107, reqid=osd.11.0:107, at_version=41'60, trim_to=0'0, roll_forward_to=41'56))", 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.938486+0000", 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.6410897200000001, 2024-09-16T04:01:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0039422750000000003, 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.938486+0000", 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.938486+0000", 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:36.669 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.938488+0000", 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8840000000000001e-06 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.938556+0000", 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8034000000000006e-05 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.938586+0000", 2024-09-16T04:01:36.670 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0849000000000002e-05 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.938608+0000", 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1943e-05 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.938827+0000", 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000218903 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:36.671 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.938852+0000", 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4822000000000001e-05 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.942307+0000", 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0034549559999999999 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.942428+0000", 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000120884 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.672 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.673 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:36.673 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:36.673 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:36.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:01:36.675 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:36.674+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:36.718 INFO:tasks.rados.rados.0.smithi188.stdout:149: done (2 left) 2024-09-16T04:01:36.718 INFO:tasks.rados.rados.0.smithi188.stdout:150: read oid 2 snap -1 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:150: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:151: read oid 8 snap -1 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:151: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:152: rollback oid 49 current snap is 10 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 49 to 5 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:148: finishing rollback tid 0 to smithi18831615-39 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:148: finishing rollback tid 1 to smithi18831615-39 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 39 v 31 (ObjNum 129 snap 0 seq_num 3906518024) dirty dne 2024-09-16T04:01:36.719 INFO:tasks.rados.rados.0.smithi188.stdout:147: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T04:01:36.754 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:01:36.763 INFO:tasks.rados.rados.0.smithi188.stdout:152: finishing rollback tid 0 to smithi18831615-49 2024-09-16T04:01:36.763 INFO:tasks.rados.rados.0.smithi188.stdout:152: finishing rollback tid 1 to smithi18831615-49 2024-09-16T04:01:36.763 INFO:tasks.rados.rados.0.smithi188.stdout:152: finishing rollback tid 2 to smithi18831615-49 2024-09-16T04:01:36.763 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 49 v 51 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T04:01:36.763 INFO:tasks.rados.rados.0.smithi188.stdout:151: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:01:36.775 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:36.770+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:36.776 INFO:tasks.rados.rados.0.smithi188.stdout:150: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:01:36.778 INFO:tasks.rados.rados.0.smithi188.stdout:147: done (4 left) 2024-09-16T04:01:36.778 INFO:tasks.rados.rados.0.smithi188.stdout:148: done (3 left) 2024-09-16T04:01:36.778 INFO:tasks.rados.rados.0.smithi188.stdout:150: done (2 left) 2024-09-16T04:01:36.778 INFO:tasks.rados.rados.0.smithi188.stdout:151: done (1 left) 2024-09-16T04:01:36.778 INFO:tasks.rados.rados.0.smithi188.stdout:152: done (0 left) 2024-09-16T04:01:36.778 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 16 current snap is 10 2024-09-16T04:01:36.778 INFO:tasks.rados.rados.0.smithi188.stdout:153: seq_num 80 ranges {1179648=589824,1769472=573440,2342912=196608} 2024-09-16T04:01:36.807 INFO:tasks.rados.rados.0.smithi188.stdout:153: writing smithi18831615-16 from 1179648 to 1769472 tid 1 2024-09-16T04:01:36.810 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:36.810 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:36.810 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:36.810 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:36.816 INFO:tasks.rados.rados.0.smithi188.stdout:153: writing smithi18831615-16 from 1769472 to 2342912 tid 2 2024-09-16T04:01:36.819 INFO:tasks.rados.rados.0.smithi188.stdout:153: writing smithi18831615-16 from 2342912 to 2539520 tid 3 2024-09-16T04:01:36.820 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 4 current snap is 10 2024-09-16T04:01:36.820 INFO:tasks.rados.rados.0.smithi188.stdout:154: seq_num 81 ranges {0=606208,606208=720896,1327104=49152} 2024-09-16T04:01:36.825 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:01:36.828 INFO:tasks.rados.rados.0.smithi188.stdout:154: writing smithi18831615-4 from 0 to 606208 tid 1 2024-09-16T04:01:36.838 INFO:tasks.rados.rados.0.smithi188.stdout:154: writing smithi18831615-4 from 606208 to 1327104 tid 2 2024-09-16T04:01:36.838 INFO:tasks.rados.rados.0.smithi188.stdout:154: writing smithi18831615-4 from 1327104 to 1376256 tid 3 2024-09-16T04:01:36.839 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 22 current snap is 10 2024-09-16T04:01:36.839 INFO:tasks.rados.rados.0.smithi188.stdout:155: seq_num 82 ranges {360448=622592,983040=475136,1458176=294912} 2024-09-16T04:01:36.849 INFO:tasks.rados.rados.0.smithi188.stdout:155: writing smithi18831615-22 from 360448 to 983040 tid 1 2024-09-16T04:01:36.853 INFO:tasks.rados.rados.0.smithi188.stdout:155: writing smithi18831615-22 from 983040 to 1458176 tid 2 2024-09-16T04:01:36.857 INFO:tasks.rados.rados.0.smithi188.stdout:155: writing smithi18831615-22 from 1458176 to 1753088 tid 3 2024-09-16T04:01:36.857 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 48 current snap is 10 2024-09-16T04:01:36.857 INFO:tasks.rados.rados.0.smithi188.stdout:153: finishing write tid 1 to smithi18831615-16 2024-09-16T04:01:36.857 INFO:tasks.rados.rados.0.smithi188.stdout:153: finishing write tid 2 to smithi18831615-16 2024-09-16T04:01:36.857 INFO:tasks.rados.rados.0.smithi188.stdout:153: finishing write tid 3 to smithi18831615-16 2024-09-16T04:01:36.857 INFO:tasks.rados.rados.0.smithi188.stdout:153: finishing write tid 4 to smithi18831615-16 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:153: finishing write tid 5 to smithi18831615-16 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:153: oid 16 updating version 0 to 38 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:153: oid 16 version 38 is already newer than 35 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:153: oid 16 version 38 is already newer than 37 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:153: oid 16 version 38 is already newer than 36 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 16 v 38 (ObjNum 80 snap 10 seq_num 80) dirty exists 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:153: left oid 16 (ObjNum 80 snap 10 seq_num 80) 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:154: finishing write tid 1 to smithi18831615-4 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:154: finishing write tid 2 to smithi18831615-4 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:154: finishing write tid 3 to smithi18831615-4 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:154: finishing write tid 4 to smithi18831615-4 2024-09-16T04:01:36.858 INFO:tasks.rados.rados.0.smithi188.stdout:156: seq_num 83 ranges {802816=638976,1441792=622592,2064384=147456} 2024-09-16T04:01:36.866 INFO:tasks.rados.rados.0.smithi188.stdout:156: writing smithi18831615-48 from 802816 to 1441792 tid 1 2024-09-16T04:01:36.871 INFO:tasks.rados.rados.0.smithi188.stdout:156: writing smithi18831615-48 from 1441792 to 2064384 tid 2 2024-09-16T04:01:36.873 INFO:tasks.rados.rados.0.smithi188.stdout:156: writing smithi18831615-48 from 2064384 to 2211840 tid 3 2024-09-16T04:01:36.873 INFO:tasks.rados.rados.0.smithi188.stdout:153: done (3 left) 2024-09-16T04:01:36.873 INFO:tasks.rados.rados.0.smithi188.stdout:157: read oid 24 snap -1 2024-09-16T04:01:36.873 INFO:tasks.rados.rados.0.smithi188.stdout:157: expect (ObjNum 70 snap 3 seq_num 70) 2024-09-16T04:01:36.873 INFO:tasks.rados.rados.0.smithi188.stdout:158: read oid 8 snap -1 2024-09-16T04:01:36.873 INFO:tasks.rados.rados.0.smithi188.stdout:158: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:01:36.873 INFO:tasks.rados.rados.0.smithi188.stdout:159: rollback oid 37 current snap is 10 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 37 to 5 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:154: finishing write tid 5 to smithi18831615-4 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:154: oid 4 updating version 0 to 69 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:154: oid 4 updating version 69 to 71 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:154: oid 4 updating version 71 to 72 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:154: oid 4 version 72 is already newer than 70 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 4 v 72 (ObjNum 81 snap 10 seq_num 81) dirty exists 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:154: left oid 4 (ObjNum 81 snap 10 seq_num 81) 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:154: done (5 left) 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:160: rmattr oid 12 current snap is 10 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:160: done (5 left) 2024-09-16T04:01:36.874 INFO:tasks.rados.rados.0.smithi188.stdout:161: snap_remove snap 3 2024-09-16T04:01:36.876 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:36.874+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:36.892 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:36.954 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:36.955 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:36.955 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:36.955 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:36.955 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:36.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:01:37.012 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:37.010+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:37.041 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:37.042 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:37.062 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:37.063 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:37.076 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:37.070+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:37.081 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:37.082 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:37.102 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:37.103 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:37.121 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:37.121 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:37.140 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:37.140 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:7 3.0s0 3:0076079f:::smithi18831615-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.845227+0000", 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.290987412, 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.017844958000000001, 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.141 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 7 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845227+0000", 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845227+0000", 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.142 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845231+0000", 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6169999999999998e-06 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845487+0000", 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025666399999999998 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845522+0000", 2024-09-16T04:01:37.143 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4304999999999998e-05 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845543+0000", 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1022999999999998e-05 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.144 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845730+0000", 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00018766 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.846074+0000", 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00034338700000000001 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.848021+0000", 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001947442 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.145 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.854006+0000", 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059846090000000001 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.862889+0000", 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0088831780000000003 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.863072+0000", 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00018307299999999999 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:8 3.0s0 3:0076079f:::smithi18831615-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.290601265999999, 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.022018801000000001, 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.147 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 8 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.148 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845615+0000", 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.967e-06 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845638+0000", 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2532e-05 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845663+0000", 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.149 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845680+0000", 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7323e-05 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.851526+0000", 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058460049999999996 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.150 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.851763+0000", 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023643399999999999 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.852890+0000", 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0011268949999999999 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.858042+0000", 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051525850000000003 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.151 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.867414+0000", 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093716719999999993 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.867632+0000", 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00021833200000000001 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.152 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:9 3.0s0 3:0076079f:::smithi18831615-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.290506013, 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.027184505000000001, 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 9 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.153 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845710+0000", 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.567e-06 2024-09-16T04:01:37.154 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845718+0000", 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7729999999999999e-06 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845740+0000", 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2048999999999998e-05 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.155 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845757+0000", 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7076e-05 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.853731+0000", 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079742270000000004 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.853981+0000", 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025017399999999998 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.156 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.868611+0000", 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014630026000000001 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.868826+0000", 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000214816 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.872893+0000", 2024-09-16T04:01:37.157 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0040667969999999996 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:66 3.9s0 3:9300f872:::smithi18831615-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.891374+0000", 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.244840699999999, 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013080227999999999, 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.158 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 66 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.891374+0000", 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.891374+0000", 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.159 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.891375+0000", 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4270000000000001e-06 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.891384+0000", 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.3559999999999993e-06 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.160 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.891405+0000", 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1226e-05 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.891420+0000", 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.473e-05 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.893501+0000", 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0020812299999999999 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.161 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.893727+0000", 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000225808 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.900076+0000", 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0063491270000000004 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.900711+0000", 2024-09-16T04:01:37.162 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00063537399999999996 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.904454+0000", 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037429500000000001 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:82 3.0s0 3:0baf0dec:::smithi18831615-28:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.902684+0000", 2024-09-16T04:01:37.163 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.233530252, 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015881954, 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 82 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.164 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.902684+0000", 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.902684+0000", 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.902687+0000", 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.39e-06 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.165 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903496+0000", 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00080918799999999996 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903528+0000", 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2308999999999998e-05 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903546+0000", 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7776999999999999e-05 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.166 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903687+0000", 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000140815 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.904006+0000", 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00031908799999999998 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.905699+0000", 2024-09-16T04:01:37.167 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001693583 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.913338+0000", 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0076389700000000001 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.918424+0000", 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0050858300000000004 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.168 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.918566+0000", 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014200399999999999 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:83 3.0s0 3:0baf0dec:::smithi18831615-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.903586+0000", 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.232628806999999, 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015491415999999999, 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.169 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 83 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903586+0000", 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.170 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903586+0000", 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903587+0000", 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.415e-06 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903597+0000", 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.747e-06 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.171 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903616+0000", 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8830000000000001e-05 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903628+0000", 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1926999999999999e-05 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.907163+0000", 2024-09-16T04:01:37.172 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0035350300000000002 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.907361+0000", 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019837399999999999 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.908297+0000", 2024-09-16T04:01:37.173 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00093636900000000005 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.913343+0000", 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0050451929999999999 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.918821+0000", 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005478241 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.174 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.919077+0000", 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025629 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:84 3.0s0 3:0baf0dec:::smithi18831615-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.23246237, 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021781656, 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.175 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 84 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.176 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903753+0000", 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1039999999999999e-06 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903759+0000", 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1380000000000004e-06 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.177 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903787+0000", 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7223999999999999e-05 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903802+0000", 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5301000000000001e-05 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.908998+0000", 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051958990000000004 2024-09-16T04:01:37.178 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.909182+0000", 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000184143 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.920281+0000", 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011099238000000001 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.920720+0000", 2024-09-16T04:01:37.179 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00043861699999999998 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.925534+0000", 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0048139919999999996 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:325 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [append 0~0] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.031723+0000", 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 11.104491747000001, 2024-09-16T04:01:37.180 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014186014, 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 325 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.031723+0000", 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.181 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.031723+0000", 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.031724+0000", 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4389999999999999e-06 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.031731+0000", 2024-09-16T04:01:37.182 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.153e-06 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.031751+0000", 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9150000000000001e-05 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.031785+0000", 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3958999999999998e-05 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.183 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039492+0000", 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0077072099999999999 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039684+0000", 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019241700000000001 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.040786+0000", 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001101894 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.184 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.043203+0000", 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0024165380000000002 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.045761+0000", 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0025583989999999998 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.045909+0000", 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014785500000000001 2024-09-16T04:01:37.185 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:327 3.0s0 3:03f9c209:::smithi18831615-40:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.246991692, 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.027429133000000001, 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.186 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 327 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.187 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889227+0000", 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8700000000000002e-06 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889241+0000", 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4606e-05 2024-09-16T04:01:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889275+0000", 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3892e-05 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889293+0000", 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7353000000000001e-05 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889422+0000", 2024-09-16T04:01:37.189 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000129799 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889435+0000", 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2167000000000001e-05 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.893120+0000", 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003685406 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.190 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.893360+0000", 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023993500000000001 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895575+0000", 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0022147410000000001 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901769+0000", 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061945739999999996 2024-09-16T04:01:37.191 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916453+0000", 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014683732999999999 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916652+0000", 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019905699999999999 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.192 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:328 3.0s0 3:03f9c209:::smithi18831615-40:head [rollback 3] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.246880736, 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.029259183000000001, 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 328 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.193 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889335+0000", 2024-09-16T04:01:37.194 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6700000000000002e-07 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889340+0000", 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8230000000000002e-06 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889352+0000", 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.234e-05 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.195 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889360+0000", 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889591+0000", 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023105599999999999 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889610+0000", 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8749e-05 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.196 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.896810+0000", 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0072003049999999997 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.897074+0000", 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00026394400000000001 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898389+0000", 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001314787 2024-09-16T04:01:37.197 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916676+0000", 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018286922000000001 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918392+0000", 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001716038 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918593+0000", 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000201289 2024-09-16T04:01:37.198 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:329 3.0s0 3:03f9c209:::smithi18831615-40:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.246841313000001, 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.031812201999999998, 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.199 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 329 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:01:37.200 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889374+0000", 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0299999999999998e-07 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889377+0000", 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2770000000000001e-06 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.201 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889387+0000", 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6260000000000007e-06 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889394+0000", 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9059999999999996e-06 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889709+0000", 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00031473299999999998 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889718+0000", 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2499999999999995e-06 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899428+0000", 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0097098859999999992 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899700+0000", 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000272558 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.203 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901104+0000", 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00140375 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920686+0000", 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019581923000000001 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920950+0000", 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00026403800000000002 2024-09-16T04:01:37.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921185+0000", 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023545200000000001 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:330 3.0s0 3:0076079f:::smithi18831615-3:head [stat,delete] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.246775395, 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.032116411999999997, 2024-09-16T04:01:37.205 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 330 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.206 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889440+0000", 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6000000000000003e-07 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889444+0000", 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:01:37.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889455+0000", 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0259e-05 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889462+0000", 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4329999999999998e-06 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889794+0000", 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000331966 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889810+0000", 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5696999999999999e-05 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902422+0000", 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012612490000000001 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.209 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902647+0000", 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00022501300000000001 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906224+0000", 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0035769059999999999 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921192+0000", 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014967644 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.210 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921363+0000", 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000171304 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921556+0000", 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019263700000000001 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.211 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s4 41/33 ECSubWrite(tid=10, reqid=osd.5.0:10, at_version=41'8, trim_to=0'0, roll_forward_to=37'6))", 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.23278438, 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.033655513999999997, 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903432+0000", 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2970000000000002e-06 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903515+0000", 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2059999999999997e-05 2024-09-16T04:01:37.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903547+0000", 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2106000000000001e-05 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903569+0000", 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1939000000000001e-05 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916774+0000", 2024-09-16T04:01:37.214 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013205099 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916797+0000", 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2809000000000001e-05 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937024+0000", 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020227256999999998 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.215 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937086+0000", 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1946999999999995e-05 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 40/33 ECSubWriteReply(tid=55, last_complete=41'32, committed=1, applied=1))", 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.903497+0000", 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.232717982, 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013183966, 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.216 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903497+0000", 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903497+0000", 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.217 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903499+0000", 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0729999999999999e-06 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903511+0000", 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2163e-05 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903540+0000", 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9128999999999998e-05 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903560+0000", 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9910000000000001e-05 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916240+0000", 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012679701999999999 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916681+0000", 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00044098899999999998 2024-09-16T04:01:37.219 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/33 ECSubWriteReply(tid=56, last_complete=41'33, committed=1, applied=1))", 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.903607+0000", 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.232607823, 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013726518, 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.220 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903607+0000", 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903607+0000", 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903608+0000", 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.15e-06 2024-09-16T04:01:37.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903617+0000", 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8000000000000004e-06 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903638+0000", 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1509e-05 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.222 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903653+0000", 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4994000000000001e-05 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917284+0000", 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013630667000000001 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917333+0000", 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.9397999999999997e-05 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.223 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/33 ECSubWriteReply(tid=56, last_complete=41'33, committed=1, applied=1))", 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.903742+0000", 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.232472980000001, 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013697704999999999, 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.224 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903742+0000", 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903742+0000", 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903743+0000", 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.883e-06 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.225 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903752+0000", 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2830000000000001e-06 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903793+0000", 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1442000000000001e-05 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903815+0000", 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1322999999999999e-05 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.226 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917393+0000", 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013578066 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917439+0000", 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.6708000000000001e-05 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.227 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/33 ECSubWriteReply(tid=56, last_complete=41'33, committed=1, applied=1))", 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.904333+0000", 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.231881429, 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013209904, 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.904333+0000", 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.228 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.904333+0000", 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.904335+0000", 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2929999999999999e-06 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.904347+0000", 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1113e-05 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.229 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.904372+0000", 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5559e-05 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.904389+0000", 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7373000000000001e-05 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917499+0000", 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013109501000000001 2024-09-16T04:01:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917543+0000", 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4064999999999999e-05 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:371 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.41793584700000003, 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018868566, 2024-09-16T04:01:37.231 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 371 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718281+0000", 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1969999999999999e-06 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718293+0000", 2024-09-16T04:01:37.233 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1636e-05 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718315+0000", 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2935999999999999e-05 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718327+0000", 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1124000000000001e-05 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.234 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718409+0000", 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2245999999999998e-05 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718416+0000", 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.657e-06 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.724610+0000", 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061949750000000001 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.235 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.724979+0000", 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00036804699999999998 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.727047+0000", 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0020681330000000002 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.735227+0000", 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0081803929999999993 2024-09-16T04:01:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.737066+0000", 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00183858 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.737147+0000", 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1642000000000002e-05 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.237 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:372 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [rollback 7] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.41785149799999999, 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021826534000000002, 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 372 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.238 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718364+0000", 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:01:37.239 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718367+0000", 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0489999999999999e-06 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718384+0000", 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7365000000000001e-05 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.240 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718398+0000", 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.358e-05 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718493+0000", 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5422999999999998e-05 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718498+0000", 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7500000000000003e-06 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.728402+0000", 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0099045539999999994 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.728508+0000", 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000105475 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.729094+0000", 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00058619300000000004 2024-09-16T04:01:37.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.736729+0000", 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0076349230000000001 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740059+0000", 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0033296939999999998 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740190+0000", 2024-09-16T04:01:37.243 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00013097299999999999 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:373 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.417803388, 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.022197585999999998, 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:37.244 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 373 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:01:37.245 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718412+0000", 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1099999999999995e-07 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718416+0000", 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.873e-06 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718427+0000", 2024-09-16T04:01:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1233000000000001e-05 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718435+0000", 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7069999999999992e-06 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718521+0000", 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6065999999999999e-05 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:37.247 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718524+0000", 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.72e-06 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732081+0000", 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0135566 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732182+0000", 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000101362 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.248 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732688+0000", 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00050549700000000004 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.736733+0000", 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004045089 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740503+0000", 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003770088 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:37.249 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740609+0000", 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010564 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:37.250 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:37.251 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:37.252 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:37.252 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:37.253 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:37.253 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:01:37.253 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:37.246+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:37.254 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:37.254 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:37.254 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:37.254 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:37.255 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:37.255 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:37.256 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:37.256 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:37.275 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:37.275 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:37.295 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:37.296 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:37.318 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:37.319 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:37.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:37.342 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:37.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:37.340+0000 7fec25e75640 0 -- 172.21.15.37:0/4019367513 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7fec20225ee0 msgr2=0x7fec202282d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:37.353 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:37.350+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:37.369 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:37.369 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:37.388 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:37.388 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:37.401 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:01:37.453 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:37.446+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:37.522 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:37.522 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:37.536 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:01:37.554 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:37.548+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:37.655 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:37.654+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:37.658 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:37.658 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:37.673 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:01:37.728 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:01:37.728 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:01:37.728 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-16T04:01:37.729 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 16 2024-09-16T04:01:37.729 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-09-16T04:01:37.756 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:37.748+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:37.765 INFO:tasks.rados.rados.0.smithi188.stdout:161: done (5 left) 2024-09-16T04:01:37.765 INFO:tasks.rados.rados.0.smithi188.stdout:162: snap_create 2024-09-16T04:01:37.765 INFO:tasks.rados.rados.0.smithi188.stdout:155: finishing write tid 1 to smithi18831615-22 2024-09-16T04:01:37.765 INFO:tasks.rados.rados.0.smithi188.stdout:155: finishing write tid 2 to smithi18831615-22 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: finishing write tid 3 to smithi18831615-22 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: finishing write tid 4 to smithi18831615-22 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: finishing write tid 5 to smithi18831615-22 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: oid 22 updating version 0 to 6 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: oid 22 updating version 6 to 8 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: oid 22 updating version 8 to 9 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: oid 22 version 9 is already newer than 7 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 22 v 9 (ObjNum 82 snap 10 seq_num 82) dirty exists 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: left oid 22 (ObjNum 82 snap 10 seq_num 82) 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:156: finishing write tid 1 to smithi18831615-48 2024-09-16T04:01:37.766 INFO:tasks.rados.rados.0.smithi188.stdout:158: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: finishing write tid 2 to smithi18831615-48 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: finishing write tid 3 to smithi18831615-48 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: finishing write tid 4 to smithi18831615-48 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:159: finishing rollback tid 0 to smithi18831615-37 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: finishing write tid 5 to smithi18831615-48 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: oid 48 updating version 0 to 76 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: oid 48 version 76 is already newer than 75 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: oid 48 updating version 76 to 77 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: oid 48 version 77 is already newer than 74 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 48 v 77 (ObjNum 83 snap 10 seq_num 83) dirty exists 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:156: left oid 48 (ObjNum 83 snap 10 seq_num 83) 2024-09-16T04:01:37.775 INFO:tasks.rados.rados.0.smithi188.stdout:159: finishing rollback tid 1 to smithi18831615-37 2024-09-16T04:01:37.776 INFO:tasks.rados.rados.0.smithi188.stdout:159: finishing rollback tid 2 to smithi18831615-37 2024-09-16T04:01:37.776 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 37 v 81 (ObjNum 59 snap 1 seq_num 59) dirty exists 2024-09-16T04:01:37.776 INFO:tasks.rados.rados.0.smithi188.stdout:157: expect (ObjNum 70 snap 3 seq_num 70) 2024-09-16T04:01:37.852 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:37.852 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:37.857 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:37.848+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:37.869 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:01:37.958 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:37.954+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:37.989 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:37.989 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:38.003 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:01:38.053 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:38.053 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:38.059 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:38.052+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:38.072 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2024-09-16T04:01:37.758915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37929523,"num_objects":50,"num_object_clones":2,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":2,"num_read":343,"num_read_kb":38894,"num_write":561,"num_write_kb":49446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":483,"ondisk_log_size":483,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":534596,"kb_used_data":77120,"kb_used_omap":24,"kb_used_meta":457063,"kb_avail":1509414844,"statfs":{"total":1546188226560,"available":1545640800256,"internally_reserved":0,"allocated":78970880,"data_stored":75984104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":468032593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":4,"num_read_kb":1393,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036059"},"pg_stats":[{"pgid":"3.f","version":"42'25","reported_seq":109,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.030143+0000","last_change":"2024-09-16T04:01:35.106687+0000","last_active":"2024-09-16T04:01:36.030143+0000","last_peered":"2024-09-16T04:01:36.030143+0000","last_clean":"2024-09-16T04:01:36.030143+0000","last_became_active":"2024-09-16T04:01:19.845422+0000","last_became_peered":"2024-09-16T04:01:19.845422+0000","last_unstale":"2024-09-16T04:01:36.030143+0000","last_undegraded":"2024-09-16T04:01:36.030143+0000","last_fullsized":"2024-09-16T04:01:36.030143+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:25.472989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010983830000000001,"stat_sum":{"num_bytes":2752512,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":6103,"num_write":36,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"3.e","version":"43'6","reported_seq":76,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:01:31.854406+0000","last_change":"2024-09-16T04:01:27.901963+0000","last_active":"2024-09-16T04:01:31.854406+0000","last_peered":"2024-09-16T04:01:31.854406+0000","last_clean":"2024-09-16T04:01:31.854406+0000","last_became_active":"2024-09-16T04:01:20.266249+0000","last_became_peered":"2024-09-16T04:01:20.266249+0000","last_unstale":"2024-09-16T04:01:31.854406+0000","last_undegraded":"2024-09-16T04:01:31.854406+0000","last_fullsized":"2024-09-16T04:01:31.854406+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.370497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055872799999999998,"stat_sum":{"num_bytes":1064960,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":2212,"num_write":6,"num_write_kb":1440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.d","version":"42'21","reported_seq":114,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.728931+0000","last_change":"2024-09-16T04:01:36.730269+0000","last_active":"2024-09-16T04:01:36.728931+0000","last_peered":"2024-09-16T04:01:36.728931+0000","last_clean":"2024-09-16T04:01:36.728931+0000","last_became_active":"2024-09-16T04:01:19.845009+0000","last_became_peered":"2024-09-16T04:01:19.845009+0000","last_unstale":"2024-09-16T04:01:36.728931+0000","last_undegraded":"2024-09-16T04:01:36.728931+0000","last_fullsized":"2024-09-16T04:01:36.728931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:42.753903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020320690000000001,"stat_sum":{"num_bytes":2523136,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":3462,"num_write":31,"num_write_kb":3600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.c","version":"42'68","reported_seq":151,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.030351+0000","last_change":"2024-09-16T04:01:35.106465+0000","last_active":"2024-09-16T04:01:36.030351+0000","last_peered":"2024-09-16T04:01:36.030351+0000","last_clean":"2024-09-16T04:01:36.030351+0000","last_became_active":"2024-09-16T04:01:20.259789+0000","last_became_peered":"2024-09-16T04:01:20.259789+0000","last_unstale":"2024-09-16T04:01:36.030351+0000","last_undegraded":"2024-09-16T04:01:36.030351+0000","last_fullsized":"2024-09-16T04:01:36.030351+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:21.868983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013342650000000001,"stat_sum":{"num_bytes":6176768,"num_objects":6,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":1021,"num_write":58,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"3.b","version":"50'31","reported_seq":122,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:35.757096+0000","last_change":"2024-09-16T04:01:35.041696+0000","last_active":"2024-09-16T04:01:35.757096+0000","last_peered":"2024-09-16T04:01:35.757096+0000","last_clean":"2024-09-16T04:01:35.757096+0000","last_became_active":"2024-09-16T04:01:20.268548+0000","last_became_peered":"2024-09-16T04:01:20.268548+0000","last_unstale":"2024-09-16T04:01:35.757096+0000","last_undegraded":"2024-09-16T04:01:35.757096+0000","last_fullsized":"2024-09-16T04:01:35.757096+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:42.304638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010915329,"stat_sum":{"num_bytes":3227648,"num_objects":8,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":29,"num_read_kb":4571,"num_write":43,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"3.a","version":"42'10","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.859162+0000","last_change":"2024-09-16T04:01:36.860939+0000","last_active":"2024-09-16T04:01:36.859162+0000","last_peered":"2024-09-16T04:01:36.859162+0000","last_clean":"2024-09-16T04:01:36.859162+0000","last_became_active":"2024-09-16T04:01:20.268001+0000","last_became_peered":"2024-09-16T04:01:20.268001+0000","last_unstale":"2024-09-16T04:01:36.859162+0000","last_undegraded":"2024-09-16T04:01:36.859162+0000","last_fullsized":"2024-09-16T04:01:36.859162+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.107177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002696253,"stat_sum":{"num_bytes":704512,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":4131,"num_write":4,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.9","version":"36'4","reported_seq":73,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:01:32.994309+0000","last_change":"2024-09-16T04:01:27.910094+0000","last_active":"2024-09-16T04:01:32.994309+0000","last_peered":"2024-09-16T04:01:32.994309+0000","last_clean":"2024-09-16T04:01:32.994309+0000","last_became_active":"2024-09-16T04:01:20.153728+0000","last_became_peered":"2024-09-16T04:01:20.153728+0000","last_unstale":"2024-09-16T04:01:32.994309+0000","last_undegraded":"2024-09-16T04:01:32.994309+0000","last_fullsized":"2024-09-16T04:01:32.994309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:39.068689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002897096,"stat_sum":{"num_bytes":1048576,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.8","version":"41'8","reported_seq":90,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.738788+0000","last_change":"2024-09-16T04:01:36.739619+0000","last_active":"2024-09-16T04:01:36.738788+0000","last_peered":"2024-09-16T04:01:36.738788+0000","last_clean":"2024-09-16T04:01:36.738788+0000","last_became_active":"2024-09-16T04:01:20.259482+0000","last_became_peered":"2024-09-16T04:01:20.259482+0000","last_unstale":"2024-09-16T04:01:36.738788+0000","last_undegraded":"2024-09-16T04:01:36.738788+0000","last_fullsized":"2024-09-16T04:01:36.738788+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.507841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001549204,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"b","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":258,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.720834+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:36.720834+0000","last_peered":"2024-09-16T04:01:36.720834+0000","last_clean":"2024-09-16T04:01:36.720834+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:36.720834+0000","last_undegraded":"2024-09-16T04:01:36.720834+0000","last_fullsized":"2024-09-16T04:01:36.720834+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":59,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.735577+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:36.735577+0000","last_peered":"2024-09-16T04:01:36.735577+0000","last_clean":"2024-09-16T04:01:36.735577+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:36.735577+0000","last_undegraded":"2024-09-16T04:01:36.735577+0000","last_fullsized":"2024-09-16T04:01:36.735577+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"42'22","reported_seq":110,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.723465+0000","last_change":"2024-09-16T04:01:36.725478+0000","last_active":"2024-09-16T04:01:36.723465+0000","last_peered":"2024-09-16T04:01:36.723465+0000","last_clean":"2024-09-16T04:01:36.723465+0000","last_became_active":"2024-09-16T04:01:20.258661+0000","last_became_peered":"2024-09-16T04:01:20.258661+0000","last_unstale":"2024-09-16T04:01:36.723465+0000","last_undegraded":"2024-09-16T04:01:36.723465+0000","last_fullsized":"2024-09-16T04:01:36.723465+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.304711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032682990000000001,"stat_sum":{"num_bytes":2342912,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":6869,"num_write":13,"num_write_kb":2864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"6","length":"1"},{"start":"b","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.857028+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:36.857028+0000","last_peered":"2024-09-16T04:01:36.857028+0000","last_clean":"2024-09-16T04:01:36.857028+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:36.857028+0000","last_undegraded":"2024-09-16T04:01:36.857028+0000","last_fullsized":"2024-09-16T04:01:36.857028+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"42'14","reported_seq":92,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:35.744999+0000","last_change":"2024-09-16T04:01:35.031588+0000","last_active":"2024-09-16T04:01:35.744999+0000","last_peered":"2024-09-16T04:01:35.744999+0000","last_clean":"2024-09-16T04:01:35.744999+0000","last_became_active":"2024-09-16T04:01:20.269891+0000","last_became_peered":"2024-09-16T04:01:20.269891+0000","last_unstale":"2024-09-16T04:01:35.744999+0000","last_undegraded":"2024-09-16T04:01:35.744999+0000","last_fullsized":"2024-09-16T04:01:35.744999+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.249786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001180572,"stat_sum":{"num_bytes":1015808,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1539,"num_write":44,"num_write_kb":992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.721557+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:36.721557+0000","last_peered":"2024-09-16T04:01:36.721557+0000","last_clean":"2024-09-16T04:01:36.721557+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:36.721557+0000","last_undegraded":"2024-09-16T04:01:36.721557+0000","last_fullsized":"2024-09-16T04:01:36.721557+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"42'47","reported_seq":116,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:01:32.994262+0000","last_change":"2024-09-16T04:01:27.939028+0000","last_active":"2024-09-16T04:01:32.994262+0000","last_peered":"2024-09-16T04:01:32.994262+0000","last_clean":"2024-09-16T04:01:32.994262+0000","last_became_active":"2024-09-16T04:01:20.257526+0000","last_became_peered":"2024-09-16T04:01:20.257526+0000","last_unstale":"2024-09-16T04:01:32.994262+0000","last_undegraded":"2024-09-16T04:01:32.994262+0000","last_fullsized":"2024-09-16T04:01:32.994262+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:20.195475+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031666692000000003,"stat_sum":{"num_bytes":3063808,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":8,"num_write":48,"num_write_kb":6688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.2","version":"30'2","reported_seq":62,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.857538+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:36.857538+0000","last_peered":"2024-09-16T04:01:36.857538+0000","last_clean":"2024-09-16T04:01:36.857538+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:36.857538+0000","last_undegraded":"2024-09-16T04:01:36.857538+0000","last_fullsized":"2024-09-16T04:01:36.857538+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"42'15","reported_seq":103,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.723280+0000","last_change":"2024-09-16T04:01:36.724416+0000","last_active":"2024-09-16T04:01:36.723280+0000","last_peered":"2024-09-16T04:01:36.723280+0000","last_clean":"2024-09-16T04:01:36.723280+0000","last_became_active":"2024-09-16T04:01:19.840099+0000","last_became_peered":"2024-09-16T04:01:19.840099+0000","last_unstale":"2024-09-16T04:01:36.723280+0000","last_undegraded":"2024-09-16T04:01:36.723280+0000","last_fullsized":"2024-09-16T04:01:36.723280+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:22.688172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028083980000000001,"stat_sum":{"num_bytes":2883584,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":29,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"6","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.5","version":"42'21","reported_seq":101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:35.745861+0000","last_change":"2024-09-16T04:01:35.296936+0000","last_active":"2024-09-16T04:01:35.745861+0000","last_peered":"2024-09-16T04:01:35.745861+0000","last_clean":"2024-09-16T04:01:35.745861+0000","last_became_active":"2024-09-16T04:01:20.258954+0000","last_became_peered":"2024-09-16T04:01:20.258954+0000","last_unstale":"2024-09-16T04:01:35.745861+0000","last_undegraded":"2024-09-16T04:01:35.745861+0000","last_fullsized":"2024-09-16T04:01:35.745861+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.697998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090668300000000003,"stat_sum":{"num_bytes":4620288,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":13,"num_write_kb":4512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:35.744891+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:35.744891+0000","last_peered":"2024-09-16T04:01:35.744891+0000","last_clean":"2024-09-16T04:01:35.744891+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:35.744891+0000","last_undegraded":"2024-09-16T04:01:35.744891+0000","last_fullsized":"2024-09-16T04:01:35.744891+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"42'22","reported_seq":112,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:35.762263+0000","last_change":"2024-09-16T04:01:34.731495+0000","last_active":"2024-09-16T04:01:35.762263+0000","last_peered":"2024-09-16T04:01:35.762263+0000","last_clean":"2024-09-16T04:01:35.762263+0000","last_became_active":"2024-09-16T04:01:20.154175+0000","last_became_peered":"2024-09-16T04:01:20.154175+0000","last_unstale":"2024-09-16T04:01:35.762263+0000","last_undegraded":"2024-09-16T04:01:35.762263+0000","last_fullsized":"2024-09-16T04:01:35.762263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:41.659818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059370600000000005,"stat_sum":{"num_bytes":2670592,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":5624,"num_write":13,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:01:31.715416+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:31.715416+0000","last_peered":"2024-09-16T04:01:31.715416+0000","last_clean":"2024-09-16T04:01:31.715416+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:31.715416+0000","last_undegraded":"2024-09-16T04:01:31.715416+0000","last_fullsized":"2024-09-16T04:01:31.715416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"36'7","reported_seq":89,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:35.745266+0000","last_change":"2024-09-16T04:01:35.031954+0000","last_active":"2024-09-16T04:01:35.745266+0000","last_peered":"2024-09-16T04:01:35.745266+0000","last_clean":"2024-09-16T04:01:35.745266+0000","last_became_active":"2024-09-16T04:01:20.266522+0000","last_became_peered":"2024-09-16T04:01:20.266522+0000","last_unstale":"2024-09-16T04:01:35.745266+0000","last_undegraded":"2024-09-16T04:01:35.745266+0000","last_fullsized":"2024-09-16T04:01:35.745266+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:39.884181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012254550000000001,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":724,"num_write":7,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":57,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:35.750543+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:35.750543+0000","last_peered":"2024-09-16T04:01:35.750543+0000","last_clean":"2024-09-16T04:01:35.750543+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:35.750543+0000","last_undegraded":"2024-09-16T04:01:35.750543+0000","last_fullsized":"2024-09-16T04:01:35.750543+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"36'2","reported_seq":90,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:01:36.730487+0000","last_change":"2024-09-16T04:01:36.731369+0000","last_active":"2024-09-16T04:01:36.730487+0000","last_peered":"2024-09-16T04:01:36.730487+0000","last_clean":"2024-09-16T04:01:36.730487+0000","last_became_active":"2024-09-16T04:01:20.267489+0000","last_became_peered":"2024-09-16T04:01:20.267489+0000","last_unstale":"2024-09-16T04:01:36.730487+0000","last_undegraded":"2024-09-16T04:01:36.730487+0000","last_fullsized":"2024-09-16T04:01:36.730487+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:39.333469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031589679999999998,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2499,"num_write":2,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:01:35.745725+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:35.745725+0000","last_peered":"2024-09-16T04:01:35.745725+0000","last_clean":"2024-09-16T04:01:35.745725+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:35.745725+0000","last_undegraded":"2024-09-16T04:01:35.745725+0000","last_fullsized":"2024-09-16T04:01:35.745725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":37339136,"num_objects":46,"num_object_clones":2,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":2,"num_read":207,"num_read_kb":38776,"num_write":356,"num_write_kb":45984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":68112384,"data_stored":70291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":323,"ondisk_log_size":323,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215115,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31088,"kb_used_data":4060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340752,"statfs":{"total":96636764160,"available":96604930048,"internally_reserved":0,"allocated":4157440,"data_stored":3873062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215115,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30916,"kb_used_data":3884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340924,"statfs":{"total":96636764160,"available":96605106176,"internally_reserved":0,"allocated":3977216,"data_stored":3686695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215116,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":6432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":6586368,"data_stored":6301990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":4560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":4669440,"data_stored":4360578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280525,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37024,"kb_used_data":5828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334816,"statfs":{"total":96636764160,"available":96598851584,"internally_reserved":0,"allocated":5967872,"data_stored":5659010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":6020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":6164480,"data_stored":5861745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280523,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29532,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342308,"statfs":{"total":96636764160,"available":96606523392,"internally_reserved":0,"allocated":2560000,"data_stored":2257265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280523,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":4376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":4481024,"data_stored":4172162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280523,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30324,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341516,"statfs":{"total":96636764160,"available":96605712384,"internally_reserved":0,"allocated":3375104,"data_stored":3062165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31536,"kb_used_data":4504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340304,"statfs":{"total":96636764160,"available":96604471296,"internally_reserved":0,"allocated":4612096,"data_stored":4309361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215115,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36508,"kb_used_data":5320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335332,"statfs":{"total":96636764160,"available":96599379968,"internally_reserved":0,"allocated":5447680,"data_stored":5159750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247819,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39608,"kb_used_data":8412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332232,"statfs":{"total":96636764160,"available":96596205568,"internally_reserved":0,"allocated":8613888,"data_stored":9286147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247820,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36444,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335396,"statfs":{"total":96636764160,"available":96599445504,"internally_reserved":0,"allocated":5373952,"data_stored":5055523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247820,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30356,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341484,"statfs":{"total":96636764160,"available":96605679616,"internally_reserved":0,"allocated":3391488,"data_stored":3162646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36420,"kb_used_data":5224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335420,"statfs":{"total":96636764160,"available":96599470080,"internally_reserved":0,"allocated":5349376,"data_stored":5841411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247818,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35340,"kb_used_data":4144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336500,"statfs":{"total":96636764160,"available":96600576000,"internally_reserved":0,"allocated":4243456,"data_stored":3934594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7950336,"data_stored":8937472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4022272,"data_stored":4022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4333568,"data_stored":4333568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2723840,"data_stored":2813952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5320704,"data_stored":5320704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4685824,"data_stored":5492736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6004736,"data_stored":6004736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3936256,"data_stored":4231168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5533696,"data_stored":5533696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1929216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3379200,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3833856,"data_stored":3833856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2723840,"data_stored":2723840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4177920,"data_stored":4177920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":3575808,"data_stored":3575808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":3981312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:38.074 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-16T04:01:38.128 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:38.129 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:38.143 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:01:38.160 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:38.158+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:38.261 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:38.258+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:38.268 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:38.268 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:38.282 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:01:38.362 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:38.358+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:38.450 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:38.451 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:38.463 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:38.458+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:38.465 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:01:38.564 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:38.560+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:38.581 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:38.581 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:38.595 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:01:38.664 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:38.662+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:38.721 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:38.721 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:38.735 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:01:38.765 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:38.760+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:38.766 INFO:tasks.rados.rados.0.smithi188.stdout:155: done (5 left) 2024-09-16T04:01:38.766 INFO:tasks.rados.rados.0.smithi188.stdout:156: done (4 left) 2024-09-16T04:01:38.766 INFO:tasks.rados.rados.0.smithi188.stdout:157: done (3 left) 2024-09-16T04:01:38.766 INFO:tasks.rados.rados.0.smithi188.stdout:158: done (2 left) 2024-09-16T04:01:38.766 INFO:tasks.rados.rados.0.smithi188.stdout:159: done (1 left) 2024-09-16T04:01:38.766 INFO:tasks.rados.rados.0.smithi188.stdout:162: done (0 left) 2024-09-16T04:01:38.766 INFO:tasks.rados.rados.0.smithi188.stdout:163: read oid 1 snap -1 2024-09-16T04:01:38.766 INFO:tasks.rados.rados.0.smithi188.stdout:163: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:01:38.767 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 37 current snap is 11 2024-09-16T04:01:38.767 INFO:tasks.rados.rados.0.smithi188.stdout:164: seq_num 84 ranges {1605632=655360,2260992=638976,2899968=131072} 2024-09-16T04:01:38.786 INFO:tasks.rados.rados.0.smithi188.stdout:164: writing smithi18831615-37 from 1605632 to 2260992 tid 1 2024-09-16T04:01:38.792 INFO:tasks.rados.rados.0.smithi188.stdout:164: writing smithi18831615-37 from 2260992 to 2899968 tid 2 2024-09-16T04:01:38.794 INFO:tasks.rados.rados.0.smithi188.stdout:164: writing smithi18831615-37 from 2899968 to 3031040 tid 3 2024-09-16T04:01:38.794 INFO:tasks.rados.rados.0.smithi188.stdout:165: setattr oid 44 current snap is 11 2024-09-16T04:01:38.794 INFO:tasks.rados.rados.0.smithi188.stdout:163: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:01:38.796 INFO:tasks.rados.rados.0.smithi188.stdout:163: done (2 left) 2024-09-16T04:01:38.797 INFO:tasks.rados.rados.0.smithi188.stdout:166: setattr oid 38 current snap is 11 2024-09-16T04:01:38.800 INFO:tasks.rados.rados.0.smithi188.stdout:167: snap_remove snap 6 2024-09-16T04:01:38.866 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:38.866+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:38.893 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:38.893 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:38.908 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:01:38.967 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:38.962+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:39.067 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:39.067 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:39.068 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:39.066+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:39.081 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:01:39.169 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:39.166+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:39.217 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:39.217 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:39.231 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:01:39.270 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:39.266+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:39.359 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:39.359 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:39.370 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:39.366+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:39.373 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:01:39.391 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:01:39.490 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:39.484+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:39.492 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:39.493 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:39.508 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:01:39.592 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:39.590+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:39.630 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:39.631 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:39.652 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:39.673 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:39.673 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:39.674 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:39.674 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:39.689 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:01:39.694 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:39.690+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:39.696 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:39.717 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:39.717 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:39.739 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:39.739 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:39.762 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:39.763 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:39.781 INFO:tasks.rados.rados.0.smithi188.stdout:167: done (3 left) 2024-09-16T04:01:39.781 INFO:tasks.rados.rados.0.smithi188.stdout:168: rollback oid 2 current snap is 11 2024-09-16T04:01:39.781 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 2 to 8 2024-09-16T04:01:39.781 INFO:tasks.rados.rados.0.smithi188.stdout:164: finishing write tid 1 to smithi18831615-37 2024-09-16T04:01:39.781 INFO:tasks.rados.rados.0.smithi188.stdout:164: finishing write tid 2 to smithi18831615-37 2024-09-16T04:01:39.781 INFO:tasks.rados.rados.0.smithi188.stdout:164: finishing write tid 3 to smithi18831615-37 2024-09-16T04:01:39.781 INFO:tasks.rados.rados.0.smithi188.stdout:164: finishing write tid 4 to smithi18831615-37 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 38 v 92 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:164: finishing write tid 5 to smithi18831615-37 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:164: oid 37 updating version 0 to 89 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:164: oid 37 version 89 is already newer than 87 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:164: oid 37 updating version 89 to 90 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:164: oid 37 version 90 is already newer than 88 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 37 v 90 (ObjNum 84 snap 11 seq_num 84) dirty exists 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:164: left oid 37 (ObjNum 84 snap 11 seq_num 84) 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 44 v 17 (ObjNum 69 snap 2 seq_num 69) dirty exists 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:164: done (3 left) 2024-09-16T04:01:39.782 INFO:tasks.rados.rados.0.smithi188.stdout:165: done (2 left) 2024-09-16T04:01:39.783 INFO:tasks.rados.rados.0.smithi188.stdout:166: done (1 left) 2024-09-16T04:01:39.783 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 7 current snap is 11 2024-09-16T04:01:39.783 INFO:tasks.rados.rados.0.smithi188.stdout:169: seq_num 85 ranges {114688=671744,786432=770048} 2024-09-16T04:01:39.787 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:39.787 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:39.791 INFO:tasks.rados.rados.0.smithi188.stdout:169: writing smithi18831615-7 from 114688 to 786432 tid 1 2024-09-16T04:01:39.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:01:39.794 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:39.790+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:39.805 INFO:tasks.rados.rados.0.smithi188.stdout:169: writing smithi18831615-7 from 786432 to 1556480 tid 2 2024-09-16T04:01:39.805 INFO:tasks.rados.rados.0.smithi188.stdout:170: copy_from oid 18 from oid 22 current snap is 11 2024-09-16T04:01:39.805 INFO:tasks.rados.rados.0.smithi188.stdout:168: finishing rollback tid 0 to smithi18831615-2 2024-09-16T04:01:39.805 INFO:tasks.rados.rados.0.smithi188.stdout:168: finishing rollback tid 1 to smithi18831615-2 2024-09-16T04:01:39.805 INFO:tasks.rados.rados.0.smithi188.stdout:168: finishing rollback tid 2 to smithi18831615-2 2024-09-16T04:01:39.805 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 2 v 96 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T04:01:39.805 INFO:tasks.rados.rados.0.smithi188.stdout:168: done (2 left) 2024-09-16T04:01:39.805 INFO:tasks.rados.rados.0.smithi188.stdout:171: snap_create 2024-09-16T04:01:39.811 INFO:tasks.rados.rados.0.smithi188.stdout:169: finishing write tid 1 to smithi18831615-7 2024-09-16T04:01:39.818 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:39.818 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:39.819 INFO:tasks.rados.rados.0.smithi188.stdout:169: finishing write tid 2 to smithi18831615-7 2024-09-16T04:01:39.820 INFO:tasks.rados.rados.0.smithi188.stdout:169: finishing write tid 3 to smithi18831615-7 2024-09-16T04:01:39.821 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:39.821 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:39.822 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:39.822 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:39.822 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:39.828 INFO:tasks.rados.rados.0.smithi188.stdout:169: finishing write tid 4 to smithi18831615-7 2024-09-16T04:01:39.828 INFO:tasks.rados.rados.0.smithi188.stdout:169: oid 7 updating version 0 to 23 2024-09-16T04:01:39.828 INFO:tasks.rados.rados.0.smithi188.stdout:169: oid 7 updating version 23 to 24 2024-09-16T04:01:39.828 INFO:tasks.rados.rados.0.smithi188.stdout:169: oid 7 updating version 24 to 25 2024-09-16T04:01:39.828 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 7 v 25 (ObjNum 85 snap 11 seq_num 85) dirty exists 2024-09-16T04:01:39.828 INFO:tasks.rados.rados.0.smithi188.stdout:169: left oid 7 (ObjNum 85 snap 11 seq_num 85) 2024-09-16T04:01:39.835 INFO:tasks.rados.rados.0.smithi188.stdout:170: finishing copy_from racing read to smithi18831615-18 2024-09-16T04:01:39.838 INFO:tasks.rados.rados.0.smithi188.stdout:170: finishing copy_from to smithi18831615-18 2024-09-16T04:01:39.838 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 18 v 9 (ObjNum 82 snap 10 seq_num 82) dirty exists 2024-09-16T04:01:39.843 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:01:39.844 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:39.844 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:39.865 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:39.865 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:39.886 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:39.886 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:39.896 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:39.888+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:39.904 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:39.948 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:39.948 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:39.966 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:39.966 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:39.966 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:39.966 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:302 3.1s0 3:845c1c62:::smithi18831615-32:head [create,setxattr dj[s][nsgcdlaO`ntnnogS`sZdZO[V\\]llWi\\gjj (11296) in=40b,setxattr qRNaVOjbUaQsk\\XsiZbOhrsOSoqX_Tqndtkb\\_Sa (1297) in=40b,setxattr `hdQPrsh[aUo`rPcbh\\tWT\\PatkS^p[P_Pdmcgdi (840) in=40b,setxattr RXicf\\^UcqO_mXh^am]mfcdttd\\mfOiPqZcomOo` (1822) in=40b,setxattr oZlXqWbdtOrc_U_m[apaV_Nr[hOOUgj`sVNpq^Rp (1473) in=40b,setxattr TrdTbcloOaiWXhjih\\mdXfYZcg\\^aOrnVOj^sf`s (10104) in=40b,setxattr XftrktNgjnZekN[]^ls\\[htgSaXkUrlVOqS[WcNt (1263) in=40b,setxattr ^qaVseQnk[`qOXVa`m]]NtQ\\psXaNYand^tQYrWt (1051) in=40b,setxattr b_e\\_i^rj`csr^q^\\e_boR\\lQNhcZg_NVqpgUQ`Q (418) in=40b,setxattr cfoPTk\\o\\Qqbi^XNOW[dbpcRb`OledY]YqcmmqUr (19016) in=40b,setxattr qjofnWtf]NipeS]\\gcjdaepnS_Pi^QgbRZd (10104) in=35b,setxattr mtVO^NRpeT]rQZYdTadneUO_^VXn^UdhZ^\\bUQiR (769) in=40b,setxattr prahkiNnm`U[rXSatop\\lk_p\\WVkdatg[XURZrsm (4560) in=40b,setxattr isZNUepUtPtRnYQaXpfelSVVR\\apSohRfqcfV`T_ (10033) in=40b,setxattr QZkproN[Zssrnra[rrbkaSXpatcreQpt`tWbOdoO (8349) in=40b,setxattr YqVPkaeiY^Tbe\\Rr\\heXkWjtOsi`iaN[TO\\^kP^] (876) in=40b,setxattr ZdfcZ]]snWVj]dZhZ]YSjVneRRoNbk`d^een^nqN (3368) in=40b,setxattr Y^XWcNfadqaYlbNpPbVRoPtgWreNtPrhWlfbbYY] (10804) in=40b,setxattr a`l_j\\^WlXY[ekSUURUhO (10104) in=21b,setxattr YgOToR[hPbQocWaoh\\h_NkYaghbpdean`\\kalgQ[ (1541) in=40b] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.917834+0000", 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.045557414999999, 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0059267190000000004, 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 302 2024-09-16T04:01:39.967 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.917834+0000", 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.917834+0000", 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.968 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.917836+0000", 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.918096+0000", 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025959600000000001 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.918115+0000", 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9735e-05 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.969 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.918125+0000", 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3130000000000004e-06 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.918203+0000", 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8347000000000001e-05 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.918345+0000", 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000142266 2024-09-16T04:01:39.970 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.919115+0000", 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00076932400000000005 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.920581+0000", 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00146638 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.923680+0000", 2024-09-16T04:01:39.971 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0030992070000000001 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.923761+0000", 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0570999999999998e-05 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.972 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:352 3.bs0 3:da0cb12f:::smithi18831615-18:head [stat,delete] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:29.922455+0000", 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.040936927000001, 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01139134, 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 352 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:39.973 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922455+0000", 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922455+0000", 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922457+0000", 2024-09-16T04:01:39.974 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7250000000000002e-06 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922472+0000", 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4278e-05 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922501+0000", 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9179999999999998e-05 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922514+0000", 2024-09-16T04:01:39.975 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2933e-05 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922600+0000", 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6774999999999995e-05 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:01:39.976 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922608+0000", 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9179999999999994e-06 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.925579+0000", 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0029709379999999998 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.925825+0000", 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00024584299999999998 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928144+0000", 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0023187759999999998 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933040+0000", 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004896119 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933767+0000", 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00072742599999999998 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933846+0000", 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8429000000000004e-05 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:362 3.bs0 3:d42ef308:::smithi18831615-39:head [append 0~0] snapc b=[b,a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746225+0000", 2024-09-16T04:01:39.979 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.2171665330000003, 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01106538, 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 362 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.980 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746225+0000", 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746225+0000", 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:39.981 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746228+0000", 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7530000000000002e-06 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746240+0000", 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2573e-05 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746278+0000", 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7583000000000003e-05 2024-09-16T04:01:39.982 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746306+0000", 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7776e-05 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746478+0000", 2024-09-16T04:01:39.983 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017186999999999999 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746699+0000", 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000221683 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748807+0000", 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021077330000000001 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.984 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:39.985 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754833+0000", 2024-09-16T04:01:39.985 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0060256429999999998 2024-09-16T04:01:39.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.985 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:39.985 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.757092+0000", 2024-09-16T04:01:39.985 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022589369999999999 2024-09-16T04:01:39.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.986 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.986 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:39.986 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.757290+0000", 2024-09-16T04:01:39.986 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000198829 2024-09-16T04:01:39.986 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:39.987 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:39.987 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:39.987 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.987 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.987 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:363 3.bs0 3:d42ef308:::smithi18831615-39:head [rollback 5] snapc b=[b,a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.2170295759999998, 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01339692, 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:39.988 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 363 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:01:39.989 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746363+0000", 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.55e-06 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746372+0000", 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9900000000000003e-06 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746396+0000", 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.357e-05 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746414+0000", 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8012999999999998e-05 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:39.991 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750473+0000", 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040591439999999998 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750763+0000", 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000290165 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752228+0000", 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014641910000000001 2024-09-16T04:01:39.992 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758077+0000", 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0058492529999999996 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758283+0000", 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020590699999999999 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.993 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.759759+0000", 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001476137 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.0s6 51/33 ECSubWrite(tid=71, reqid=client.4646.0:371, at_version=51'49, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.727934+0000", 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.2354577330000001, 2024-09-16T04:01:39.994 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0074818259999999996, 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.727934+0000", 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.727934+0000", 2024-09-16T04:01:39.995 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:39.996 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.996 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.996 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:39.996 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.727934+0000", 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1699999999999997e-07 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.727994+0000", 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9406e-05 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.728093+0000", 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8723999999999999e-05 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.997 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.728142+0000", 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9775999999999999e-05 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.728308+0000", 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016542799999999999 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.728333+0000", 2024-09-16T04:01:39.998 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4746e-05 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.735318+0000", 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069858840000000004 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:39.999 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.735416+0000", 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7145000000000006e-05 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.0s6 51/33 ECSubWrite(tid=72, reqid=client.4646.0:372, at_version=51'50, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.729635+0000", 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.233756557, 2024-09-16T04:01:40.000 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0097369429999999996, 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729635+0000", 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729635+0000", 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.001 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729636+0000", 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3790000000000001e-06 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729682+0000", 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5626e-05 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729714+0000", 2024-09-16T04:01:40.002 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2376000000000003e-05 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729737+0000", 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2203000000000001e-05 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730081+0000", 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034408399999999998 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.003 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730106+0000", 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5347e-05 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.739281+0000", 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0091746369999999994 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.004 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.739372+0000", 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1291000000000004e-05 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 51/33 ECSubWriteReply(tid=89, last_complete=51'33, committed=1, applied=1))", 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.730095+0000", 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.2332963989999999, 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0068228960000000002, 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.005 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730095+0000", 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730095+0000", 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.006 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730098+0000", 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.61e-06 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730110+0000", 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2734e-05 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.007 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730139+0000", 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8272e-05 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730245+0000", 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000106279 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.008 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732068+0000", 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0018233730000000001 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.736918+0000", 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048496279999999999 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.0s6 51/33 ECSubWrite(tid=73, reqid=client.4646.0:373, at_version=51'51, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:40.009 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.732393+0000", 2024-09-16T04:01:40.010 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.2309988129999998, 2024-09-16T04:01:40.010 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.007126701, 2024-09-16T04:01:40.010 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.010 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.010 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732393+0000", 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732393+0000", 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.011 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732394+0000", 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.81e-06 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732438+0000", 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3022000000000002e-05 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732470+0000", 2024-09-16T04:01:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2882999999999997e-05 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732499+0000", 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8336000000000001e-05 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732717+0000", 2024-09-16T04:01:40.013 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021781700000000001 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.732741+0000", 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4669e-05 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.739380+0000", 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0066391289999999997 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.014 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.739519+0000", 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000139035 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.0s6 51/33 ECSubWrite(tid=74, reqid=client.4646.0:373, at_version=0'0, trim_to=0'0, roll_forward_to=51'51))", 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.740484+0000", 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.2229078050000002, 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0052837639999999998, 2024-09-16T04:01:40.015 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.740484+0000", 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.740484+0000", 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.016 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.740485+0000", 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7630000000000001e-06 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.740506+0000", 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0426000000000001e-05 2024-09-16T04:01:40.017 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.740555+0000", 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9032000000000001e-05 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.740570+0000", 2024-09-16T04:01:40.018 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5347000000000001e-05 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.740709+0000", 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013889499999999999 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.740731+0000", 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2116e-05 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.019 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.745702+0000", 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00497066 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.745767+0000", 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5524999999999994e-05 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.020 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:374 3.bs0 3:de226c4e:::smithi18831615-16:head [append 0~589824] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:40.021 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.806551+0000", 2024-09-16T04:01:40.021 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.156840893, 2024-09-16T04:01:40.021 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0096058089999999999, 2024-09-16T04:01:40.021 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.021 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.021 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:40.021 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 374 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.806551+0000", 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.806551+0000", 2024-09-16T04:01:40.022 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.806554+0000", 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5329999999999999e-06 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807550+0000", 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00099578300000000008 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.023 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807589+0000", 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9446000000000001e-05 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807609+0000", 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9582999999999999e-05 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807760+0000", 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015121399999999999 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.024 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807938+0000", 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017827399999999999 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.809819+0000", 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001880912 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.815226+0000", 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0054064530000000003 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.025 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.815343+0000", 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000116888 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.816156+0000", 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081372300000000003 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.026 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:375 3.bs0 3:de226c4e:::smithi18831615-16:head [append 0~573440] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.815913+0000", 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.1474787960000001, 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0080350830000000002, 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 375 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.027 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.815913+0000", 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.815913+0000", 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.028 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.815915+0000", 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9860000000000001e-06 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.816652+0000", 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00073705099999999996 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.029 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.816668+0000", 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6458000000000001e-05 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.816678+0000", 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4350000000000003e-06 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.816865+0000", 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000187794 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.817111+0000", 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00024560599999999999 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818562+0000", 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014509379999999999 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.031 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.032 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823246+0000", 2024-09-16T04:01:40.032 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0046845059999999997 2024-09-16T04:01:40.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.032 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:40.032 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823837+0000", 2024-09-16T04:01:40.032 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059032900000000001 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823948+0000", 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011098 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:376 3.bs0 3:de226c4e:::smithi18831615-16:head [append 0~196608] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:40.033 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.819189+0000", 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.1442028180000001, 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0066361939999999998, 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.034 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 376 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819189+0000", 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819189+0000", 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.035 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.036 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819190+0000", 2024-09-16T04:01:40.036 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7770000000000001e-06 2024-09-16T04:01:40.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.036 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.036 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.036 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819397+0000", 2024-09-16T04:01:40.036 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020685899999999999 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819416+0000", 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8981e-05 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.037 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819429+0000", 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2459e-05 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.820812+0000", 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00138291 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.820892+0000", 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0747000000000005e-05 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821454+0000", 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00056119899999999996 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823598+0000", 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021444319999999999 2024-09-16T04:01:40.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.825762+0000", 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021644500000000001 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.825825+0000", 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2379999999999996e-05 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:377 3.bs0 3:de226c4e:::smithi18831615-16:head [setxattr _header (57) in=7b] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.819453+0000", 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.1439384420000001, 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0091754439999999996, 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.041 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 377 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819453+0000", 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.043 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.043 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819453+0000", 2024-09-16T04:01:40.043 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.043 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.043 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819454+0000", 2024-09-16T04:01:40.043 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.082e-06 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819459+0000", 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2220000000000001e-06 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819473+0000", 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4106e-05 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819482+0000", 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9339999999999995e-06 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822129+0000", 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0026468389999999998 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822264+0000", 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013523300000000001 2024-09-16T04:01:40.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822808+0000", 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00054320799999999999 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828019+0000", 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0052111170000000004 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:40.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828531+0000", 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051255599999999995 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828628+0000", 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7146999999999997e-05 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.047 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:378 3.bs0 3:de226c4e:::smithi18831615-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.1438935890000002, 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.013087019, 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 378 2024-09-16T04:01:40.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:01:40.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.050 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.050 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819499+0000", 2024-09-16T04:01:40.050 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1710000000000001e-06 2024-09-16T04:01:40.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.051 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819503+0000", 2024-09-16T04:01:40.051 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9319999999999998e-06 2024-09-16T04:01:40.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.051 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819519+0000", 2024-09-16T04:01:40.051 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5939000000000001e-05 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819528+0000", 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9390000000000003e-06 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823625+0000", 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040971779999999999 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:40.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823698+0000", 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3417999999999994e-05 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.829121+0000", 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0054222879999999999 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.829385+0000", 2024-09-16T04:01:40.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026470200000000001 2024-09-16T04:01:40.054 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.054 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.054 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.054 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.832585+0000", 2024-09-16T04:01:40.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031994520000000002 2024-09-16T04:01:40.054 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.054 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds6 54/33 ECSubWrite(tid=33, reqid=client.4646.0:419, at_version=54'23, trim_to=0'0, roll_forward_to=42'21))", 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.795187+0000", 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.168204248, 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049390119999999996, 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.055 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.795187+0000", 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.795187+0000", 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.795188+0000", 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6800000000000002e-07 2024-09-16T04:01:40.056 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.795921+0000", 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00073343300000000002 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.795958+0000", 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7184999999999997e-05 2024-09-16T04:01:40.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.795970+0000", 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2079000000000001e-05 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.796173+0000", 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020268900000000001 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.796187+0000", 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4053000000000001e-05 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.800088+0000", 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0039007569999999999 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.800126+0000", 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8547999999999998e-05 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.059 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.060 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.060 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.060 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.060 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.0s6 54/33 ECSubWrite(tid=86, reqid=osd.14.0:86, at_version=54'55, trim_to=0'0, roll_forward_to=52'53))", 2024-09-16T04:01:40.060 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.802606+0000", 2024-09-16T04:01:40.060 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.160785074, 2024-09-16T04:01:40.060 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0096979389999999992, 2024-09-16T04:01:40.060 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802606+0000", 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.061 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802606+0000", 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802607+0000", 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9799999999999999e-07 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802628+0000", 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1413e-05 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802641+0000", 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2778e-05 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802651+0000", 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9229999999999997e-06 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802725+0000", 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4374000000000003e-05 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802737+0000", 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.167e-05 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812215+0000", 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0094780650000000008 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812304+0000", 2024-09-16T04:01:40.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9418000000000003e-05 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:423 3.bs0 3:da0cb12f:::smithi18831615-18:head [copy-from ver 9] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.15881598799999999, 2024-09-16T04:01:40.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.032968130999999998, 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 423 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.066 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804577+0000", 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9759999999999998e-06 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804585+0000", 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6890000000000004e-06 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804602+0000", 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6912999999999999e-05 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804611+0000", 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7600000000000008e-06 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804668+0000", 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7521e-05 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804745+0000", 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6137999999999995e-05 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.832170+0000", 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.027425393999999999 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834467+0000", 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022970199999999999 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.837427+0000", 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002960074 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.837544+0000", 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011664599999999999 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.071 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:424 3.bs0 3:da0cb12f:::smithi18831615-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.15876210399999999, 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.029561375000000001, 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 424 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804630+0000", 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6400000000000002e-07 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804633+0000", 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0800000000000002e-06 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804642+0000", 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2520000000000005e-06 2024-09-16T04:01:40.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804650+0000", 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3590000000000002e-06 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804908+0000", 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025827199999999998 2024-09-16T04:01:40.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.076 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:01:40.076 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804983+0000", 2024-09-16T04:01:40.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5432000000000006e-05 2024-09-16T04:01:40.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.833934+0000", 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.028950975 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834053+0000", 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000118553 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834191+0000", 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013788799999999999 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds6 54/33 ECSubWrite(tid=34, reqid=client.4646.0:420, at_version=54'24, trim_to=0'0, roll_forward_to=42'21))", 2024-09-16T04:01:40.078 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.808861+0000", 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.154530784, 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0090378139999999999, 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.808861+0000", 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.808861+0000", 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.808862+0000", 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809209+0000", 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034700900000000001 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809230+0000", 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1203e-05 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809239+0000", 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3360000000000001e-06 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809356+0000", 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000117198 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809366+0000", 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6139999999999998e-06 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.817855+0000", 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0084890469999999996 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.817899+0000", 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3531e-05 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds6 54/33 ECSubWrite(tid=35, reqid=client.4646.0:421, at_version=54'25, trim_to=0'0, roll_forward_to=54'23))", 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.812333+0000", 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.15105803100000001, 2024-09-16T04:01:40.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0055849810000000001, 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812333+0000", 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812333+0000", 2024-09-16T04:01:40.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.086 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812334+0000", 2024-09-16T04:01:40.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4900000000000005e-07 2024-09-16T04:01:40.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812352+0000", 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7974000000000001e-05 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812365+0000", 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3144e-05 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812374+0000", 2024-09-16T04:01:40.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7420000000000003e-06 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812435+0000", 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0739999999999998e-05 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812441+0000", 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1630000000000001e-06 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.088 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.817903+0000", 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.005462204 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.817918+0000", 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5265e-05 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.089 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:40.091 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:01:40.092 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:40.093 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:40.084+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:40.192 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:40.190+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:40.253 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:40.253 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:40.253 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:40.253 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:40.267 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:01:40.293 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:40.290+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:40.394 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:40.394+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:40.396 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:40.396 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:40.396 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:40.396 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:40.396 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:40.411 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:01:40.495 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:40.490+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:40.540 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.888440+0000", 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.647654171999999, 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0041107130000000002, 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.541 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888440+0000", 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888440+0000", 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888441+0000", 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1800000000000004e-07 2024-09-16T04:01:40.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.889248+0000", 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00080720499999999997 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.889263+0000", 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4419999999999999e-05 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.889274+0000", 2024-09-16T04:01:40.543 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1745e-05 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.889377+0000", 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010234900000000001 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.889388+0000", 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0916e-05 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.544 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892491+0000", 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031037759999999999 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892551+0000", 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9383999999999999e-05 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:296 3.4s0 3:2e4a2a62:::smithi18831615-31:head [rollback 3] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:40.545 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.886877+0000", 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.649216814000001, 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0090995360000000001, 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 296 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.546 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886877+0000", 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886877+0000", 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886879+0000", 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.547 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886888+0000", 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4889999999999995e-06 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886909+0000", 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1540999999999999e-05 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886923+0000", 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3715e-05 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886997+0000", 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4371999999999999e-05 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.887004+0000", 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2430000000000002e-06 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.889751+0000", 2024-09-16T04:01:40.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027470530000000002 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.889862+0000", 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011124900000000001 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.895977+0000", 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0061149539999999997 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.550 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 40/33 ECSubWrite(tid=85, reqid=client.4646.0:297, at_version=40'45, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.906005+0000", 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.630089427, 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040996369999999997, 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906005+0000", 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.551 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906005+0000", 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906005+0000", 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.82e-07 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906330+0000", 2024-09-16T04:01:40.552 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00032465400000000001 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906353+0000", 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2626e-05 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906371+0000", 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8620000000000001e-05 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.553 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906449+0000", 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7416000000000006e-05 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906456+0000", 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5660000000000001e-06 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.910050+0000", 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035935160000000002 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.554 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.910104+0000", 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4756999999999998e-05 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s1 40/33 ECSubWrite(tid=20, reqid=client.4646.0:316, at_version=40'15, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.022171+0000", 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.513922892, 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004073859, 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.555 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022171+0000", 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022171+0000", 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.556 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022172+0000", 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.13e-06 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022197+0000", 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4664999999999998e-05 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022217+0000", 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9624e-05 2024-09-16T04:01:40.557 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022228+0000", 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1753000000000001e-05 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022294+0000", 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5061000000000006e-05 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.022304+0000", 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0376e-05 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.026197+0000", 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0038930779999999999 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.026245+0000", 2024-09-16T04:01:40.559 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8171999999999999e-05 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds5 41/33 ECSubWrite(tid=28, reqid=osd.7.0:28, at_version=41'17, trim_to=0'0, roll_forward_to=38'15))", 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.905492+0000", 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.630601915, 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0041881100000000001, 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.905492+0000", 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.905492+0000", 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.561 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.905492+0000", 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.29e-07 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.905515+0000", 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2949e-05 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.905530+0000", 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4391e-05 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.562 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.905543+0000", 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2964e-05 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.905593+0000", 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0204999999999998e-05 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.905599+0000", 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.604e-06 2024-09-16T04:01:40.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.909645+0000", 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004045936 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.909680+0000", 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5831999999999998e-05 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s1 41/33 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=41'19, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.906259+0000", 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.629835141000001, 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049730959999999998, 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906259+0000", 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906259+0000", 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906260+0000", 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1299999999999999e-07 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.566 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906288+0000", 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7923999999999999e-05 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906311+0000", 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2833e-05 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906321+0000", 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0827e-05 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906396+0000", 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4733000000000004e-05 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906408+0000", 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1701e-05 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.911183+0000", 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0047755810000000001 2024-09-16T04:01:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.911232+0000", 2024-09-16T04:01:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8683999999999997e-05 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 41/33 ECSubWrite(tid=101, reqid=osd.11.0:101, at_version=41'54, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.912686+0000", 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.62340805, 2024-09-16T04:01:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0043890759999999996, 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.912686+0000", 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.912686+0000", 2024-09-16T04:01:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.912687+0000", 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3300000000000003e-07 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.912705+0000", 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8760999999999999e-05 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.912720+0000", 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4466e-05 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.912731+0000", 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1122e-05 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.912779+0000", 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8551999999999999e-05 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.912785+0000", 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7810000000000002e-06 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.917047+0000", 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004261702 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.917075+0000", 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8259000000000002e-05 2024-09-16T04:01:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs4 41/33 ECSubWrite(tid=100, reqid=osd.11.0:100, at_version=41'21, trim_to=0'0, roll_forward_to=40'19))", 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.913152+0000", 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.622942117999999, 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0065104020000000002, 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913152+0000", 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913152+0000", 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913153+0000", 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913169+0000", 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6121999999999999e-05 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913180+0000", 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1076e-05 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913189+0000", 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2439999999999999e-06 2024-09-16T04:01:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913234+0000", 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4867999999999998e-05 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913241+0000", 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3929999999999996e-06 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919631+0000", 2024-09-16T04:01:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0063907 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919662+0000", 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1167999999999997e-05 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.915250+0000", 2024-09-16T04:01:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.620843949999999, 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0044905700000000002, 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915250+0000", 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915250+0000", 2024-09-16T04:01:40.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915251+0000", 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0399999999999995e-07 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915267+0000", 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6067999999999999e-05 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915278+0000", 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1412e-05 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915284+0000", 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9460000000000003e-06 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915337+0000", 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2473999999999999e-05 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915342+0000", 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6500000000000001e-06 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919721+0000", 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0043782259999999998 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919741+0000", 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0089999999999999e-05 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.585 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds5 41/33 ECSubWrite(tid=30, reqid=osd.7.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=41'19))", 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.917972+0000", 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.618122136, 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0042641670000000001, 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.586 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.917972+0000", 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.917972+0000", 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.917973+0000", 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.917991+0000", 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7815000000000001e-05 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.918002+0000", 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.187e-05 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.918008+0000", 2024-09-16T04:01:40.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7520000000000002e-06 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.918069+0000", 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1019e-05 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.918076+0000", 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0520000000000004e-06 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922193+0000", 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0041171910000000001 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922236+0000", 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2771000000000001e-05 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:331 3.4s0 3:2a076d46:::smithi18831615-43:head [copy-from ver 17] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.613642865999999, 2024-09-16T04:01:40.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.031262501999999998, 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 331 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:01:40.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922453+0000", 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.341e-06 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922458+0000", 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4070000000000001e-06 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922470+0000", 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2363e-05 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922476+0000", 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5380000000000002e-06 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922556+0000", 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0481000000000004e-05 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922641+0000", 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4303999999999998e-05 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.946336+0000", 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.023695544999999998 2024-09-16T04:01:40.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953385+0000", 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0070484689999999999 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953631+0000", 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00024578499999999999 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953714+0000", 2024-09-16T04:01:40.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3269000000000005e-05 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs4 41/33 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=41'23, trim_to=0'0, roll_forward_to=40'19))", 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.922570+0000", 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.61352377, 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051287989999999999, 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.596 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922570+0000", 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922570+0000", 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922571+0000", 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8400000000000003e-07 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922597+0000", 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5839000000000001e-05 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922610+0000", 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3305999999999999e-05 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.598 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922620+0000", 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1649999999999995e-06 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922688+0000", 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8880999999999993e-05 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922694+0000", 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9089999999999997e-06 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.927653+0000", 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049587800000000003 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.927699+0000", 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6035000000000002e-05 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:332 3.4s0 3:2a076d46:::smithi18831615-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:01:40.600 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.613416635, 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.026354305000000001, 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 332 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5900000000000005e-07 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922697+0000", 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8867000000000001e-05 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922710+0000", 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2591e-05 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922720+0000", 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0353000000000001e-05 2024-09-16T04:01:40.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922817+0000", 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6675000000000005e-05 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922866+0000", 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9060000000000001e-05 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.948791+0000", 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.025925143000000001 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.948905+0000", 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011362300000000001 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949032+0000", 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012703399999999999 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.605 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 41/33 ECSubWrite(tid=110, reqid=osd.11.0:109, at_version=0'0, trim_to=0'0, roll_forward_to=41'62))", 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.949009+0000", 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.587084770000001, 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.005080574, 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949009+0000", 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949009+0000", 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949010+0000", 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2200000000000003e-07 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949025+0000", 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4595e-05 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949037+0000", 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2404e-05 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949049+0000", 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1673e-05 2024-09-16T04:01:40.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949099+0000", 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0135e-05 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949108+0000", 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7139999999999999e-06 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:40.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.954059+0000", 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049509330000000002 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.954090+0000", 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1398e-05 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:359 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:40.610 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.7898937029999997, 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015151255000000001, 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 359 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746203+0000", 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6529999999999998e-06 2024-09-16T04:01:40.612 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746217+0000", 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3946e-05 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746250+0000", 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3269999999999998e-05 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746268+0000", 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.804e-05 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746410+0000", 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014167300000000001 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746432+0000", 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1999999999999999e-05 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750758+0000", 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004325527 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750901+0000", 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000143361 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.761352+0000", 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010450785000000001 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:360 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.7897828389999999, 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016051205999999998, 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.616 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 360 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746313+0000", 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3200000000000001e-06 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746326+0000", 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2874000000000001e-05 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746352+0000", 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6395000000000001e-05 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746372+0000", 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9584000000000001e-05 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746569+0000", 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019797599999999999 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746579+0000", 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:01:40.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753659+0000", 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0070800610000000003 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753897+0000", 2024-09-16T04:01:40.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023741600000000001 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.762363+0000", 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0084659499999999999 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.621 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:361 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.7896918340000001, 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016661775, 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 361 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.622 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746403+0000", 2024-09-16T04:01:40.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746412+0000", 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3229999999999998e-06 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746433+0000", 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1484e-05 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746450+0000", 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6252e-05 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746636+0000", 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018645399999999999 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746645+0000", 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9560000000000003e-06 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754590+0000", 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0079450430000000006 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754769+0000", 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017916000000000001 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.763064+0000", 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0082949830000000006 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.626 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.4s0 50/33 ECSubRead(tid=38, to_read={3:20e031d1:::smithi18831615-29:head=0,118784,0}, subchunks={3:20e031d1:::smithi18831615-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.782622892, 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049784879999999997, 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.627 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755777+0000", 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0023053320000000002 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755801+0000", 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4729999999999999e-05 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.756189+0000", 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00038755899999999999 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758450+0000", 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022608670000000002 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.4s0 50/33 ECSubRead(tid=39, to_read={3:20e031d1:::smithi18831615-29:head=0,118784,0}, subchunks={3:20e031d1:::smithi18831615-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:40.630 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.754485+0000", 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.7816090999999998, 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0043294900000000001, 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754485+0000", 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.631 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754485+0000", 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754485+0000", 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754485+0000", 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755862+0000", 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013773220000000001 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755883+0000", 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0656e-05 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758687+0000", 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0028036179999999999 2024-09-16T04:01:40.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758815+0000", 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012789400000000001 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.4s0 50/33 ECSubRead(tid=40, to_read={3:20e031d1:::smithi18831615-29:head=0,118784,0}, subchunks={3:20e031d1:::smithi18831615-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.755331+0000", 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.7807632350000002, 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00450301, 2024-09-16T04:01:40.634 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755331+0000", 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755331+0000", 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755331+0000", 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755331+0000", 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755929+0000", 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059779199999999998 2024-09-16T04:01:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755950+0000", 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1350000000000001e-05 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.759572+0000", 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036219989999999999 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.759834+0000", 2024-09-16T04:01:40.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026186900000000002 2024-09-16T04:01:40.638 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.638 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.638 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.638 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:40.638 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:40.638 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:40.639 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:01:40.640 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:40.638+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:40.741 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:40.738+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:40.773 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:40.773 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:40.773 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:40.773 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:40.777 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-09-16T04:01:40.783 INFO:tasks.rados.rados.0.smithi188.stdout:169: done (2 left) 2024-09-16T04:01:40.783 INFO:tasks.rados.rados.0.smithi188.stdout:170: done (1 left) 2024-09-16T04:01:40.783 INFO:tasks.rados.rados.0.smithi188.stdout:171: done (0 left) 2024-09-16T04:01:40.783 INFO:tasks.rados.rados.0.smithi188.stdout:172: snap_remove snap 7 2024-09-16T04:01:40.792 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:01:40.845 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:40.840+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:40.919 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:40.919 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:40.919 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:40.919 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:40.920 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:40.933 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:01:40.942 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:40.942+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:41.043 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:41.036+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.217354934999999, 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0044144730000000004, 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.109 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888568+0000", 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.67e-07 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890276+0000", 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0017087719999999999 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890297+0000", 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0384e-05 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890309+0000", 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1827999999999999e-05 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890402+0000", 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3830000000000001e-05 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890412+0000", 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8050000000000001e-06 2024-09-16T04:01:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892961+0000", 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0025483979999999999 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892982+0000", 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1189e-05 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 39/33 ECSubWrite(tid=81, reqid=client.4646.0:292, at_version=39'42, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.891848+0000", 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.214074525000001, 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003440556, 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891848+0000", 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891848+0000", 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891849+0000", 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9400000000000004e-07 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891935+0000", 2024-09-16T04:01:41.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6601999999999995e-05 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891949+0000", 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3878e-05 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.891959+0000", 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7399999999999999e-06 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892017+0000", 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8057999999999999e-05 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892026+0000", 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4630000000000007e-06 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.895246+0000", 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003219965 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.895288+0000", 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1956000000000003e-05 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 40/33 ECSubWrite(tid=85, reqid=client.4646.0:297, at_version=40'45, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.906746+0000", 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.199176360999999, 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004320089, 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.118 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906746+0000", 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906746+0000", 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906746+0000", 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9200000000000001e-07 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906973+0000", 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022612000000000001 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.907008+0000", 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5404000000000002e-05 2024-09-16T04:01:41.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.907038+0000", 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.044e-05 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.907149+0000", 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000110197 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.907164+0000", 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5673000000000001e-05 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.911000+0000", 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003836023 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.911066+0000", 2024-09-16T04:01:41.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5740000000000004e-05 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 41/33 ECSubWrite(tid=71, reqid=osd.4.0:69, at_version=0'0, trim_to=0'0, roll_forward_to=41'20))", 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.906848+0000", 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.199074798, 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0025816390000000002, 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.123 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906848+0000", 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906848+0000", 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906848+0000", 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.16e-07 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906860+0000", 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2058999999999999e-05 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906877+0000", 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6409000000000002e-05 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906888+0000", 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0772e-05 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906961+0000", 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3825000000000006e-05 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.906971+0000", 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7059999999999999e-06 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.909350+0000", 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0023784520000000001 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.909429+0000", 2024-09-16T04:01:41.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9599999999999997e-05 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 41/33 ECSubWrite(tid=101, reqid=osd.11.0:101, at_version=41'54, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.913687+0000", 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.192235712, 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0030601930000000001, 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.128 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913687+0000", 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913687+0000", 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913687+0000", 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8699999999999995e-07 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913718+0000", 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0345000000000002e-05 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913741+0000", 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3819999999999999e-05 2024-09-16T04:01:41.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913757+0000", 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5335000000000001e-05 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913903+0000", 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014641400000000001 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913914+0000", 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0835999999999999e-05 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.916709+0000", 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027947919999999999 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.916747+0000", 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8064e-05 2024-09-16T04:01:41.132 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.915429+0000", 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.190493467, 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0039867210000000004, 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.133 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915429+0000", 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915429+0000", 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915430+0000", 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5600000000000004e-07 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915455+0000", 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5551e-05 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915472+0000", 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7147e-05 2024-09-16T04:01:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915482+0000", 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7440000000000002e-06 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915540+0000", 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7469000000000002e-05 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915545+0000", 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8730000000000004e-06 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919371+0000", 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0038254890000000001 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919416+0000", 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4592000000000003e-05 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 41/33 ECSubWrite(tid=105, reqid=osd.11.0:102, at_version=0'0, trim_to=0'0, roll_forward_to=41'56))", 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.934403+0000", 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.171518827, 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0029160599999999998, 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.934403+0000", 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.934403+0000", 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.934409+0000", 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6489999999999996e-06 2024-09-16T04:01:41.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.934427+0000", 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7527e-05 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.934451+0000", 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4732999999999999e-05 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.934464+0000", 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2312e-05 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.934546+0000", 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1798000000000005e-05 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.934557+0000", 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1093e-05 2024-09-16T04:01:41.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.937278+0000", 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027214380000000001 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.937320+0000", 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1510000000000001e-05 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 41/33 ECSubWrite(tid=106, reqid=osd.11.0:106, at_version=41'58, trim_to=0'0, roll_forward_to=41'56))", 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.936547+0000", 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.169375090000001, 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0026922169999999998, 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.936547+0000", 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.936547+0000", 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.936548+0000", 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7599999999999996e-07 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.936586+0000", 2024-09-16T04:01:41.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7837999999999999e-05 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.936600+0000", 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4219999999999999e-05 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.936617+0000", 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6481999999999999e-05 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.936927+0000", 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00031038000000000002 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.936936+0000", 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4059999999999994e-06 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.939209+0000", 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022725190000000002 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.939239+0000", 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0596000000000003e-05 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 44/33 ECSubWrite(tid=82, reqid=client.4646.0:352, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:29.928253+0000", 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.177669454, 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050133139999999996, 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.147 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928253+0000", 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928253+0000", 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928254+0000", 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928308+0000", 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4018999999999999e-05 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928433+0000", 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012573099999999999 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928459+0000", 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5752999999999999e-05 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928608+0000", 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014835099999999999 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928619+0000", 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1736000000000001e-05 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933215+0000", 2024-09-16T04:01:41.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045959820000000002 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933266+0000", 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0881999999999998e-05 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 44/33 ECSubWrite(tid=83, reqid=client.4646.0:352, at_version=0'0, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:29.934164+0000", 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.171757848, 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040574779999999998, 2024-09-16T04:01:41.151 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934164+0000", 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934164+0000", 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934166+0000", 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8759999999999999e-06 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934195+0000", 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8197000000000001e-05 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934243+0000", 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8899000000000001e-05 2024-09-16T04:01:41.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934267+0000", 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3853e-05 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934410+0000", 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000142387 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934445+0000", 2024-09-16T04:01:41.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5272999999999999e-05 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.938182+0000", 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0037373789999999999 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.938222+0000", 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9613999999999997e-05 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 49/33 ECSubWrite(tid=84, reqid=osd.4.0:84, at_version=49'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.033687+0000", 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.0722357149999997, 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048717830000000002, 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033687+0000", 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033687+0000", 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033687+0000", 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033747+0000", 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033793+0000", 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6202999999999998e-05 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033820+0000", 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6461999999999998e-05 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033999+0000", 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017919400000000001 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.034024+0000", 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5279999999999999e-05 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.038490+0000", 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00446592 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.160 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.038558+0000", 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8306000000000006e-05 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 50/33 ECSubWrite(tid=86, reqid=client.4646.0:362, at_version=50'30, trim_to=0'0, roll_forward_to=49'28))", 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.748953+0000", 2024-09-16T04:01:41.161 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.3569697520000004, 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040299580000000002, 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748953+0000", 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748953+0000", 2024-09-16T04:01:41.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748953+0000", 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2600000000000002e-07 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749016+0000", 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2515999999999996e-05 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749052+0000", 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6189000000000002e-05 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749078+0000", 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6106e-05 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749280+0000", 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020179 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749305+0000", 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5242999999999999e-05 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752877+0000", 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035716670000000002 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752983+0000", 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000105721 2024-09-16T04:01:41.165 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 50/33 ECSubWrite(tid=87, reqid=client.4646.0:363, at_version=50'31, trim_to=0'0, roll_forward_to=49'28))", 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.753041+0000", 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.3528816810000004, 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027431809999999999, 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.166 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753041+0000", 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753041+0000", 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753042+0000", 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.796e-06 2024-09-16T04:01:41.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753089+0000", 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6179e-05 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753120+0000", 2024-09-16T04:01:41.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1763000000000001e-05 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753143+0000", 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2756e-05 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753858+0000", 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00071521400000000004 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753884+0000", 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5446999999999999e-05 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755690+0000", 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001806247 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755784+0000", 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3778999999999998e-05 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=89, reqid=osd.4.0:89, at_version=51'33, trim_to=0'0, roll_forward_to=50'31))", 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.719262+0000", 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.386660429, 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027401679999999999, 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.171 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.719262+0000", 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.719262+0000", 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.719263+0000", 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6000000000000003e-07 2024-09-16T04:01:41.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.719345+0000", 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2446000000000003e-05 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.719379+0000", 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3788999999999998e-05 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.719405+0000", 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5780000000000001e-05 2024-09-16T04:01:41.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.719599+0000", 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019461000000000001 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.719631+0000", 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1705999999999998e-05 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.721906+0000", 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022753790000000001 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.722002+0000", 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5798000000000007e-05 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=90, reqid=osd.4.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=51'33))", 2024-09-16T04:01:41.175 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.733755+0000", 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.3721668769999997, 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045050910000000001, 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733755+0000", 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.176 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733755+0000", 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733757+0000", 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9630000000000001e-06 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733795+0000", 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7523e-05 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733828+0000", 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3495999999999999e-05 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733852+0000", 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3946000000000001e-05 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.734111+0000", 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025884900000000001 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.734137+0000", 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6137000000000001e-05 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.738170+0000", 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040330920000000003 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.738261+0000", 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0085000000000002e-05 2024-09-16T04:01:41.179 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=93, reqid=client.4646.0:375, at_version=51'36, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.818799+0000", 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.2871235570000001, 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0046750480000000002, 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.180 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818799+0000", 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818799+0000", 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818800+0000", 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.133e-06 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.181 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818967+0000", 2024-09-16T04:01:41.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016715400000000001 2024-09-16T04:01:41.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818985+0000", 2024-09-16T04:01:41.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7580000000000001e-05 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818994+0000", 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819109+0000", 2024-09-16T04:01:41.185 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000115531 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819123+0000", 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4173e-05 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823440+0000", 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004317052 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823474+0000", 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3364000000000001e-05 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=94, reqid=client.4646.0:376, at_version=51'37, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.821592+0000", 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.2843303419999996, 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0038958069999999998, 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.187 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821592+0000", 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821592+0000", 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821593+0000", 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5999999999999991e-07 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821673+0000", 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0075999999999997e-05 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821700+0000", 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7154000000000001e-05 2024-09-16T04:01:41.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821715+0000", 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5167999999999999e-05 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821800+0000", 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4556999999999997e-05 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.190 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821814+0000", 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3793e-05 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.825444+0000", 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036301319999999999 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.825488+0000", 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3967000000000001e-05 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.191 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=95, reqid=client.4646.0:377, at_version=51'38, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.822903+0000", 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.2830191639999997, 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053964039999999996, 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.192 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822903+0000", 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822903+0000", 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822905+0000", 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7659999999999999e-06 2024-09-16T04:01:41.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822972+0000", 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6878999999999994e-05 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823001+0000", 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9346e-05 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823022+0000", 2024-09-16T04:01:41.194 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.12e-05 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823123+0000", 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000100784 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823136+0000", 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2624e-05 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.195 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828258+0000", 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051225280000000003 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828300+0000", 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1276999999999998e-05 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/33 ECSubWrite(tid=135, reqid=client.4646.0:389, at_version=51'74, trim_to=0'0, roll_forward_to=51'72))", 2024-09-16T04:01:41.196 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.868821+0000", 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.2371010079999998, 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031114609999999998, 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.868821+0000", 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.197 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.868821+0000", 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.868824+0000", 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5739999999999998e-06 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.869257+0000", 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000433485 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.869283+0000", 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.597e-05 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.869297+0000", 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.384e-05 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.869433+0000", 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000136122 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.199 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.869459+0000", 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6012000000000001e-05 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.871882+0000", 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0024230430000000002 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.871933+0000", 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0414999999999998e-05 2024-09-16T04:01:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/33 ECSubWrite(tid=140, reqid=client.4646.0:391, at_version=51'76, trim_to=0'0, roll_forward_to=51'74))", 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.876768+0000", 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.2291541239999999, 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031208479999999999, 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.201 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.876768+0000", 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.876768+0000", 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.876769+0000", 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0330000000000001e-06 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.876816+0000", 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6292999999999999e-05 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.876834+0000", 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8797999999999999e-05 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.876847+0000", 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2807e-05 2024-09-16T04:01:41.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.877006+0000", 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000158777 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.877031+0000", 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4876999999999999e-05 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.879835+0000", 2024-09-16T04:01:41.204 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0028046730000000001 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.879889+0000", 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.359e-05 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:41.205 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:41.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:01:41.208 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:41.202+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:41.308 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:41.302+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:41.343 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:41.343 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:41.343 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:41.343 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:41.357 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:01:41.409 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:41.406+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:41.510 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:41.506+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:41.530 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:41.530 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:41.530 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:41.530 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:41.530 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:41.545 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:01:41.611 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:41.610+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:41.712 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:41.710+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 44/33 ECSubWrite(tid=82, reqid=client.4646.0:352, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.928412+0000", 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.785126789, 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0050467949999999998, 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928412+0000", 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928412+0000", 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928413+0000", 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0499999999999999e-06 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928470+0000", 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7218e-05 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928506+0000", 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5803000000000003e-05 2024-09-16T04:01:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928531+0000", 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5142000000000001e-05 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929620+0000", 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010886800000000001 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929658+0000", 2024-09-16T04:01:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7771999999999997e-05 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.933359+0000", 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003701307 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.933459+0000", 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9822999999999996e-05 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 44/33 ECSubWrite(tid=83, reqid=client.4646.0:352, at_version=0'0, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.934498+0000", 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.779041018999999, 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025193559999999999, 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934498+0000", 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934498+0000", 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934500+0000", 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.897e-06 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934522+0000", 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2659000000000001e-05 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934553+0000", 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1031999999999997e-05 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934569+0000", 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6019999999999999e-05 2024-09-16T04:01:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934729+0000", 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000159438 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934762+0000", 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2842999999999997e-05 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.936917+0000", 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021552899999999998 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.937017+0000", 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000100177 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 49/33 ECSubWrite(tid=84, reqid=osd.4.0:84, at_version=49'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:35.033892+0000", 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.6796467660000003, 2024-09-16T04:01:41.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025354459999999998, 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.033892+0000", 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.033892+0000", 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.033893+0000", 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.23e-07 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.033953+0000", 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9988000000000002e-05 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.033991+0000", 2024-09-16T04:01:41.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8525999999999997e-05 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034017+0000", 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6115999999999999e-05 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034207+0000", 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000189655 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034236+0000", 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9196e-05 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.036366+0000", 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021298300000000001 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.036428+0000", 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1412000000000001e-05 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.731 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 50/33 ECSubWrite(tid=86, reqid=client.4646.0:362, at_version=50'30, trim_to=0'0, roll_forward_to=49'28))", 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:35.749022+0000", 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.9645169999999998, 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026027310000000001, 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749022+0000", 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749022+0000", 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749023+0000", 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749082+0000", 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9864999999999997e-05 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749107+0000", 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4859999999999999e-05 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749132+0000", 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4295999999999999e-05 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749318+0000", 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186338 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749335+0000", 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7223000000000001e-05 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.751540+0000", 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022048250000000001 2024-09-16T04:01:41.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.751625+0000", 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4604000000000005e-05 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 51/33 ECSubWrite(tid=89, reqid=osd.4.0:89, at_version=51'33, trim_to=0'0, roll_forward_to=50'31))", 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.719637+0000", 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.9939022519999998, 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0092214389999999997, 2024-09-16T04:01:41.736 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719637+0000", 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719637+0000", 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719637+0000", 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.13e-07 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719695+0000", 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7408000000000003e-05 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719732+0000", 2024-09-16T04:01:41.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7579000000000001e-05 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719757+0000", 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4856999999999999e-05 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719919+0000", 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016154200000000001 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719936+0000", 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7595999999999999e-05 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.728769+0000", 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0088326029999999996 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.728858+0000", 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9240999999999995e-05 2024-09-16T04:01:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 51/33 ECSubWrite(tid=71, reqid=client.4646.0:371, at_version=51'49, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.726849+0000", 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.9866901810000002, 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046563330000000003, 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726849+0000", 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726849+0000", 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726849+0000", 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726904+0000", 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4698999999999999e-05 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726935+0000", 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1259e-05 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726959+0000", 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3856e-05 2024-09-16T04:01:41.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727186+0000", 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022679699999999999 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727215+0000", 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.923e-05 2024-09-16T04:01:41.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.731406+0000", 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004190839 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.731505+0000", 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9170000000000001e-05 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 51/33 ECSubWrite(tid=72, reqid=client.4646.0:372, at_version=51'50, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.729189+0000", 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.9843498080000002, 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00388223, 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729189+0000", 2024-09-16T04:01:41.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729189+0000", 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729191+0000", 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.72e-06 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729249+0000", 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8424999999999997e-05 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729281+0000", 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1466e-05 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729298+0000", 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7346e-05 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729454+0000", 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015631299999999999 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729481+0000", 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6469e-05 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.732927+0000", 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034464579999999999 2024-09-16T04:01:41.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.733071+0000", 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014403300000000001 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 51/33 ECSubWrite(tid=90, reqid=osd.4.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=51'33))", 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.734192+0000", 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.9793472469999998, 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025276780000000002, 2024-09-16T04:01:41.750 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734192+0000", 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734192+0000", 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734194+0000", 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.877e-06 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734220+0000", 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6516000000000002e-05 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734254+0000", 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4479e-05 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734279+0000", 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4142999999999999e-05 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734435+0000", 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000156489 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734461+0000", 2024-09-16T04:01:41.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5553000000000001e-05 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736597+0000", 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021358179999999998 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736719+0000", 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122803 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 51/33 ECSubWrite(tid=91, reqid=client.4646.0:374, at_version=51'35, trim_to=0'0, roll_forward_to=51'33))", 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.810542+0000", 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.9029969170000003, 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035567580000000001, 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.810542+0000", 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.810542+0000", 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.810544+0000", 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8330000000000001e-06 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811015+0000", 2024-09-16T04:01:41.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00047113800000000001 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811047+0000", 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2416000000000003e-05 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811073+0000", 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5599e-05 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811290+0000", 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021699800000000001 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811308+0000", 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8502999999999999e-05 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.813995+0000", 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026869939999999998 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.814099+0000", 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010327699999999999 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 51/33 ECSubWrite(tid=93, reqid=client.4646.0:375, at_version=51'36, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.819131+0000", 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.8944079279999997, 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00257792, 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.759 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819131+0000", 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819131+0000", 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819133+0000", 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.59e-06 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819497+0000", 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036486999999999999 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819528+0000", 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0233e-05 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819554+0000", 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6786000000000001e-05 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819690+0000", 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013543600000000001 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819705+0000", 2024-09-16T04:01:41.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5007e-05 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.821607+0000", 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001902124 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.821709+0000", 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010187399999999999 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.763 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 51/33 ECSubWrite(tid=75, reqid=client.4646.0:384, at_version=51'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.851681+0000", 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.8618580619999996, 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003433555, 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.851681+0000", 2024-09-16T04:01:41.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.851681+0000", 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.851683+0000", 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.852182+0000", 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00049975400000000004 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.852213+0000", 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0521999999999997e-05 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.852229+0000", 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5926e-05 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.852367+0000", 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138114 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.852386+0000", 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9171999999999999e-05 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.855048+0000", 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026613689999999998 2024-09-16T04:01:41.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.855114+0000", 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6810999999999994e-05 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 51/33 ECSubWrite(tid=76, reqid=client.4646.0:385, at_version=51'7, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.856672+0000", 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.8568665840000005, 2024-09-16T04:01:41.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036080069999999999, 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.856672+0000", 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.856672+0000", 2024-09-16T04:01:41.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.856678+0000", 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6699999999999999e-06 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.857254+0000", 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057574199999999996 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.857286+0000", 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2292000000000002e-05 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.857309+0000", 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2727000000000001e-05 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.857490+0000", 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018109499999999999 2024-09-16T04:01:41.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.857516+0000", 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5996e-05 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.860187+0000", 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026716819999999999 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.860280+0000", 2024-09-16T04:01:41.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2802999999999999e-05 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 51/33 ECSubWrite(tid=77, reqid=client.4646.0:386, at_version=51'8, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.858261+0000", 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.8552781380000001, 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033116550000000002, 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.773 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.858261+0000", 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.858261+0000", 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.858262+0000", 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6330000000000001e-06 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.858462+0000", 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000199175 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.858488+0000", 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6213000000000001e-05 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.858502+0000", 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4236000000000001e-05 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.858861+0000", 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035867499999999998 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.858876+0000", 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5199e-05 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.861502+0000", 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002625699 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.861572+0000", 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0825e-05 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/33 ECSubWrite(tid=85, reqid=osd.14.0:85, at_version=54'13, trim_to=0'0, roll_forward_to=52'11))", 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.799788+0000", 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.9137511599999999, 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035245070000000001, 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.799788+0000", 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.799788+0000", 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.799788+0000", 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4000000000000001e-07 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.799826+0000", 2024-09-16T04:01:41.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7178999999999998e-05 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.799850+0000", 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4280000000000001e-05 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.799866+0000", 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6016e-05 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800039+0000", 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000172783 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800066+0000", 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7319000000000001e-05 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.803213+0000", 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003147113 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.803312+0000", 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9177000000000003e-05 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 54/33 ECSubWrite(tid=86, reqid=osd.14.0:86, at_version=54'55, trim_to=0'0, roll_forward_to=52'53))", 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.800790+0000", 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.9127491910000001, 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045216520000000001, 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800790+0000", 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800790+0000", 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800792+0000", 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.967e-06 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800847+0000", 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5627999999999997e-05 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800874+0000", 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6659999999999999e-05 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800894+0000", 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9939e-05 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.801124+0000", 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022965799999999999 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.801160+0000", 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6075999999999998e-05 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.805209+0000", 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040491329999999999 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.805311+0000", 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010259099999999999 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 54/33 ECSubWrite(tid=101, reqid=osd.4.0:101, at_version=54'44, trim_to=0'0, roll_forward_to=52'42))", 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.802102+0000", 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.9114368269999999, 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0047704360000000003, 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802102+0000", 2024-09-16T04:01:41.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802102+0000", 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802103+0000", 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802157+0000", 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4005999999999998e-05 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802190+0000", 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3009000000000002e-05 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802212+0000", 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1855999999999999e-05 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802392+0000", 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018071499999999999 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802418+0000", 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5516000000000001e-05 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.806782+0000", 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0043635790000000002 2024-09-16T04:01:41.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.806872+0000", 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0977999999999995e-05 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 54/33 ECSubWrite(tid=102, reqid=osd.4.0:102, at_version=54'46, trim_to=0'0, roll_forward_to=52'42))", 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.802262+0000", 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.911277187, 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053342440000000001, 2024-09-16T04:01:41.793 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802262+0000", 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802262+0000", 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802263+0000", 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7600000000000006e-07 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802316+0000", 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2855999999999997e-05 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802342+0000", 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6166e-05 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802365+0000", 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3272999999999999e-05 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.804138+0000", 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017730090000000001 2024-09-16T04:01:41.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.804163+0000", 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5199e-05 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.807527+0000", 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033641840000000001 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.807596+0000", 2024-09-16T04:01:41.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8580999999999999e-05 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 54/33 ECSubWrite(tid=87, reqid=osd.14.0:86, at_version=0'0, trim_to=0'0, roll_forward_to=54'55))", 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.814164+0000", 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.8993750309999999, 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002590602, 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814164+0000", 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814164+0000", 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814166+0000", 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.736e-06 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814188+0000", 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2636000000000001e-05 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814246+0000", 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7772000000000002e-05 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814272+0000", 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6156000000000002e-05 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814437+0000", 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016447599999999999 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814465+0000", 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8328999999999999e-05 2024-09-16T04:01:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.816662+0000", 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021970589999999999 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.816754+0000", 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2437999999999998e-05 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/33 ECSubWrite(tid=88, reqid=osd.14.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=54'13))", 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.814850+0000", 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.898689268, 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037884220000000001, 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814850+0000", 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814850+0000", 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814851+0000", 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814879+0000", 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7293000000000001e-05 2024-09-16T04:01:41.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814910+0000", 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1461999999999998e-05 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814933+0000", 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3332999999999998e-05 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.815104+0000", 2024-09-16T04:01:41.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170142 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.815127+0000", 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3603000000000001e-05 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.818548+0000", 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034209959999999999 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.818638+0000", 2024-09-16T04:01:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9896999999999997e-05 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 54/33 ECSubWrite(tid=104, reqid=client.4646.0:423, at_version=54'47, trim_to=0'0, roll_forward_to=54'46))", 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.832982+0000", 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.8805570440000001, 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004648386, 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:41.807 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.832982+0000", 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.832982+0000", 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.832984+0000", 2024-09-16T04:01:41.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.68e-06 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834372+0000", 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013879929999999999 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834403+0000", 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:41.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834419+0000", 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5766000000000001e-05 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834538+0000", 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011849399999999999 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834554+0000", 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-05 2024-09-16T04:01:41.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.837545+0000", 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029916029999999998 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.837630+0000", 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4890999999999997e-05 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:41.811 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:41.812 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:41.812 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:172: done (0 left) 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:173: read oid 34 snap 5 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:173: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 45 current snap is 12 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:174: seq_num 86 ranges {753664=688128,1441792=540672,1982464=229376} 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:174: writing smithi18831615-45 from 753664 to 1441792 tid 1 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:174: writing smithi18831615-45 from 1441792 to 1982464 tid 2 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:174: writing smithi18831615-45 from 1982464 to 2211840 tid 3 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:175: snap_create 2024-09-16T04:01:41.813 INFO:tasks.rados.rados.0.smithi188.stdout:173: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:01:41.815 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:41.814+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:41.822 INFO:tasks.rados.rados.0.smithi188.stdout:174: finishing write tid 1 to smithi18831615-45 2024-09-16T04:01:41.822 INFO:tasks.rados.rados.0.smithi188.stdout:174: finishing write tid 2 to smithi18831615-45 2024-09-16T04:01:41.822 INFO:tasks.rados.rados.0.smithi188.stdout:174: finishing write tid 3 to smithi18831615-45 2024-09-16T04:01:41.822 INFO:tasks.rados.rados.0.smithi188.stdout:174: finishing write tid 4 to smithi18831615-45 2024-09-16T04:01:41.822 INFO:tasks.rados.rados.0.smithi188.stdout:174: finishing write tid 5 to smithi18831615-45 2024-09-16T04:01:41.822 INFO:tasks.rados.rados.0.smithi188.stdout:174: oid 45 updating version 0 to 25 2024-09-16T04:01:41.822 INFO:tasks.rados.rados.0.smithi188.stdout:174: oid 45 version 25 is already newer than 21 2024-09-16T04:01:41.822 INFO:tasks.rados.rados.0.smithi188.stdout:174: oid 45 version 25 is already newer than 24 2024-09-16T04:01:41.823 INFO:tasks.rados.rados.0.smithi188.stdout:174: oid 45 updating version 25 to 26 2024-09-16T04:01:41.823 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 45 v 26 (ObjNum 86 snap 12 seq_num 86) dirty exists 2024-09-16T04:01:41.823 INFO:tasks.rados.rados.0.smithi188.stdout:174: left oid 45 (ObjNum 86 snap 12 seq_num 86) 2024-09-16T04:01:41.915 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:41.910+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:41.943 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:41.943 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:41.943 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:41.943 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:41.958 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:01:42.017 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:42.014+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:42.082 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:42.083 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:42.083 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:42.083 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:42.083 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:42.093 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:42.097 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:01:42.137 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:42.132+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:42.218 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:42.218+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:42.227 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:42.227 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:42.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:42.227 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4371.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.506396838000001, 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01704027, 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4371", 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.37:0/2503621667", 2024-09-16T04:01:42.228 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716696+0000", 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716711+0000", 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4676e-05 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716743+0000", 2024-09-16T04:01:42.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1835000000000003e-05 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716769+0000", 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6327999999999999e-05 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716973+0000", 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020410900000000001 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:01:42.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.717006+0000", 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3084999999999999e-05 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.724942+0000", 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0079355829999999995 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.724964+0000", 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1804999999999999e-05 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730286+0000", 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053221689999999999 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730641+0000", 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035484999999999998 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730957+0000", 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0003165 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.732250+0000", 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001293039 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733535+0000", 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001284477 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733616+0000", 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0964999999999995e-05 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733727+0000", 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011134 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:24.888075+0000", 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.335008796, 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052425479999999997, 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.235 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888075+0000", 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888075+0000", 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888075+0000", 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.84e-07 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888746+0000", 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067078999999999999 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888787+0000", 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0757999999999998e-05 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888807+0000", 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0775999999999999e-05 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.888982+0000", 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000174277 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.889009+0000", 2024-09-16T04:01:42.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6923e-05 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.893256+0000", 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042475059999999999 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:24.893317+0000", 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1134000000000001e-05 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 40/33 ECSubWrite(tid=85, reqid=client.4646.0:297, at_version=40'45, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:25.906320+0000", 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.316763339000001, 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0028227819999999998, 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.906320+0000", 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.906320+0000", 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.906320+0000", 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.22e-07 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.906666+0000", 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00034543300000000002 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.906682+0000", 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6600999999999999e-05 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.906693+0000", 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.101e-05 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.906791+0000", 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7925999999999997e-05 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.906807+0000", 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.539e-05 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.909089+0000", 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022823409999999998 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.909143+0000", 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3859000000000001e-05 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 41/33 ECSubWrite(tid=55, reqid=client.4646.0:327, at_version=41'32, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:42.244 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.895777+0000", 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.327306093000001, 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034661729999999999, 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.895777+0000", 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.895777+0000", 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.895778+0000", 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.895813+0000", 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4959000000000002e-05 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.895846+0000", 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3346000000000002e-05 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.895871+0000", 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5262e-05 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.896057+0000", 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186064 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.896085+0000", 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7613000000000001e-05 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.899208+0000", 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031232629999999998 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.899244+0000", 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5296000000000002e-05 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 41/33 ECSubWrite(tid=56, reqid=client.4646.0:328, at_version=41'33, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.898496+0000", 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.324587521, 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037485729999999998, 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.249 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898496+0000", 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898496+0000", 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898498+0000", 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7659999999999999e-06 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898540+0000", 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2178999999999997e-05 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898569+0000", 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8759e-05 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1466000000000001e-05 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898845+0000", 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025504599999999998 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898855+0000", 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6949999999999995e-06 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.902129+0000", 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032743529999999998 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.902245+0000", 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000115309 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.253 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s3 41/33 ECSubWrite(tid=68, reqid=osd.4.0:68, at_version=41'10, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.898788+0000", 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.324295894, 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062726839999999997, 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898788+0000", 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898788+0000", 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898789+0000", 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898844+0000", 2024-09-16T04:01:42.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5358000000000001e-05 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898874+0000", 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9742000000000001e-05 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898895+0000", 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1503000000000001e-05 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.899119+0000", 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022357900000000001 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.899143+0000", 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3969999999999999e-05 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904971+0000", 2024-09-16T04:01:42.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058285979999999999 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.905060+0000", 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9050999999999998e-05 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s3 41/33 ECSubWrite(tid=70, reqid=osd.4.0:70, at_version=41'12, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:01:42.258 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.901187+0000", 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.321896156999999, 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0059245130000000002, 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901187+0000", 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901187+0000", 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901189+0000", 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8500000000000001e-06 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901259+0000", 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9641999999999995e-05 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901293+0000", 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4245000000000002e-05 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901338+0000", 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4712999999999997e-05 2024-09-16T04:01:42.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901460+0000", 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012246800000000001 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901485+0000", 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4946000000000001e-05 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907023+0000", 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055373590000000004 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907112+0000", 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9289999999999994e-05 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.263 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 41/33 ECSubWrite(tid=57, reqid=client.4646.0:329, at_version=41'34, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.904054+0000", 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.319029910999999, 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039146160000000001, 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904054+0000", 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904054+0000", 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904056+0000", 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3e-06 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904102+0000", 2024-09-16T04:01:42.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6511999999999998e-05 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904133+0000", 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0673000000000003e-05 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904155+0000", 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1535999999999998e-05 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904399+0000", 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024424499999999998 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.904425+0000", 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5729e-05 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907897+0000", 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034720839999999998 2024-09-16T04:01:42.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907968+0000", 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1537000000000003e-05 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 51/33 ECSubWrite(tid=71, reqid=client.4646.0:371, at_version=51'49, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.727120+0000", 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.4959633390000002, 2024-09-16T04:01:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0038171120000000001, 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727120+0000", 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727120+0000", 2024-09-16T04:01:42.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727121+0000", 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727159+0000", 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8439000000000002e-05 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727182+0000", 2024-09-16T04:01:42.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2657e-05 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727198+0000", 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6365999999999999e-05 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727348+0000", 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014955900000000001 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727373+0000", 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5347e-05 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.730856+0000", 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034830999999999998 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.730937+0000", 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1273999999999995e-05 2024-09-16T04:01:42.272 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 51/33 ECSubWrite(tid=72, reqid=client.4646.0:372, at_version=51'50, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.729532+0000", 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.4935518139999999, 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031269330000000001, 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729532+0000", 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729532+0000", 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729533+0000", 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7269999999999999e-06 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729570+0000", 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6445000000000001e-05 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729612+0000", 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2138999999999998e-05 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729629+0000", 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7260000000000001e-05 2024-09-16T04:01:42.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729783+0000", 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015415000000000001 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.729809+0000", 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5185e-05 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.732511+0000", 2024-09-16T04:01:42.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027027980000000002 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.732659+0000", 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000147229 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 51/33 ECSubWrite(tid=73, reqid=client.4646.0:373, at_version=51'51, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.736536+0000", 2024-09-16T04:01:42.277 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.4865478269999999, 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003553358, 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736536+0000", 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736536+0000", 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736538+0000", 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.153e-06 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736582+0000", 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3899000000000001e-05 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736621+0000", 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9035999999999997e-05 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736645+0000", 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3728999999999999e-05 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.737457+0000", 2024-09-16T04:01:42.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00081204499999999995 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.737481+0000", 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4797e-05 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.739993+0000", 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002512125 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.740089+0000", 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5574000000000003e-05 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 51/33 ECSubWrite(tid=135, reqid=client.4646.0:389, at_version=51'74, trim_to=0'0, roll_forward_to=51'72))", 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.869171+0000", 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3539125890000001, 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029458750000000001, 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.282 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.869171+0000", 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.869171+0000", 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.869173+0000", 2024-09-16T04:01:42.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9870000000000002e-06 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.869638+0000", 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00046470499999999999 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.869669+0000", 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1165999999999999e-05 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.869684+0000", 2024-09-16T04:01:42.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5536000000000002e-05 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.869816+0000", 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013140600000000001 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.869830+0000", 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4600000000000001e-05 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.872022+0000", 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021917400000000002 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.872117+0000", 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4734999999999993e-05 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.286 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 51/33 ECSubWrite(tid=136, reqid=client.4646.0:390, at_version=51'75, trim_to=0'0, roll_forward_to=51'72))", 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.874896+0000", 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3481875429999999, 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002769919, 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.874896+0000", 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.874896+0000", 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.874898+0000", 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6530000000000001e-06 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.875214+0000", 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031634600000000001 2024-09-16T04:01:42.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.875245+0000", 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1035999999999999e-05 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.875260+0000", 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5169e-05 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.875420+0000", 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015991000000000001 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.875449+0000", 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8799e-05 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.877560+0000", 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021111120000000001 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.877666+0000", 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105894 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 51/33 ECSubWrite(tid=147, reqid=client.4646.0:400, at_version=51'79, trim_to=0'0, roll_forward_to=51'77))", 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.887408+0000", 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3356754679999998, 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029086519999999999, 2024-09-16T04:01:42.291 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.292 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.292 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.292 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.292 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.292 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.292 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.887408+0000", 2024-09-16T04:01:42.292 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.292 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.887408+0000", 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.887410+0000", 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7540000000000001e-06 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.887448+0000", 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8025000000000003e-05 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.887478+0000", 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0392999999999999e-05 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.887493+0000", 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5206e-05 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.294 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.887629+0000", 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013514000000000001 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.887657+0000", 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8782e-05 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.890225+0000", 2024-09-16T04:01:42.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025676079999999999 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.890317+0000", 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1743999999999994e-05 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 51/33 ECSubWrite(tid=149, reqid=client.4646.0:402, at_version=51'81, trim_to=0'0, roll_forward_to=51'77))", 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.890614+0000", 2024-09-16T04:01:42.296 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3324696769999997, 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029073520000000002, 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.890614+0000", 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.297 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.890614+0000", 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.890615+0000", 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.890637+0000", 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2626999999999999e-05 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.298 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.890651+0000", 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3539e-05 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.890661+0000", 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7569999999999999e-06 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.891821+0000", 2024-09-16T04:01:42.299 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011599959999999999 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.891836+0000", 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5590000000000002e-05 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.893465+0000", 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016293480000000001 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.300 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.893521+0000", 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5761999999999999e-05 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 54/33 ECSubWrite(tid=162, reqid=client.4646.0:417, at_version=54'95, trim_to=0'0, roll_forward_to=53'92))", 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.789912+0000", 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 2.4331714760000001, 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002697133, 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.789912+0000", 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.789912+0000", 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.789914+0000", 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.789951+0000", 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7616e-05 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.789974+0000", 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2495999999999999e-05 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.789988+0000", 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4535999999999999e-05 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.790093+0000", 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010449400000000001 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.790108+0000", 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5500000000000001e-05 2024-09-16T04:01:42.305 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.305 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.305 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.305 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.792544+0000", 2024-09-16T04:01:42.305 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00243509 2024-09-16T04:01:42.305 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.305 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.792609+0000", 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5650999999999999e-05 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 54/33 ECSubWrite(tid=86, reqid=osd.14.0:86, at_version=54'55, trim_to=0'0, roll_forward_to=52'53))", 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.800935+0000", 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 2.422148081, 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039929539999999999, 2024-09-16T04:01:42.306 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800935+0000", 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800935+0000", 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.307 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800937+0000", 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.296e-06 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800985+0000", 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8269000000000002e-05 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.801014+0000", 2024-09-16T04:01:42.308 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8598999999999998e-05 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.801038+0000", 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4426e-05 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.801181+0000", 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142774 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.309 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.801204+0000", 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3652e-05 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.804823+0000", 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036185179999999998 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.804928+0000", 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010542 2024-09-16T04:01:42.310 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 54/33 ECSubWrite(tid=166, reqid=osd.11.0:165, at_version=0'0, trim_to=0'0, roll_forward_to=54'100))", 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.814616+0000", 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 2.408467323, 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0028763460000000001, 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.311 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814616+0000", 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814616+0000", 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814618+0000", 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1799999999999999e-06 2024-09-16T04:01:42.312 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814650+0000", 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1126e-05 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814694+0000", 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4252000000000003e-05 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814718+0000", 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4695999999999999e-05 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814920+0000", 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000201175 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.814948+0000", 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7892e-05 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.314 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.817389+0000", 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00244174 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.817493+0000", 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103285 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.3s5 56/33 ECSubWrite(tid=213, reqid=client.4646.0:430, at_version=56'21, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:41.757503+0000", 2024-09-16T04:01:42.315 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.46558020999999999, 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040464899999999998, 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.757503+0000", 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.316 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.757503+0000", 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.757504+0000", 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6800000000000002e-07 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.757651+0000", 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014703599999999999 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.317 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.757667+0000", 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5871000000000001e-05 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.757677+0000", 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0686000000000001e-05 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.757817+0000", 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000139426 2024-09-16T04:01:42.318 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.757846+0000", 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9526000000000002e-05 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.761511+0000", 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036650139999999999 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.761550+0000", 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8662999999999999e-05 2024-09-16T04:01:42.319 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.3s5 56/33 ECSubWrite(tid=214, reqid=osd.13.0:214, at_version=56'23, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:41.760044+0000", 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.46303919700000001, 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029019710000000002, 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:42.320 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.760044+0000", 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.760044+0000", 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.760045+0000", 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9299999999999997e-07 2024-09-16T04:01:42.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.760063+0000", 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7927999999999998e-05 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.760075+0000", 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1796e-05 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.760082+0000", 2024-09-16T04:01:42.322 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2289999999999997e-06 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.760211+0000", 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012888 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.760237+0000", 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6080000000000001e-05 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.323 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.762818+0000", 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025811010000000001 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.762946+0000", 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000128164 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:42.324 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:42.325 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:42.326 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:42.326 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:42.327 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:42.327 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:01:42.328 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:42.328 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:42.328 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:42.328 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:42.329 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:42.322+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:42.338 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:42.338 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:42.360 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:42.360 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:42.383 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:42.383 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:42.406 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:42.406 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:42.426 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:42.420+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:42.429 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:42.429 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:42.452 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:42.452 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:42.460 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:42.460 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:42.474 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:01:42.477 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:42.477 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:42.501 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:42.501 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:42.524 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:42.527 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:42.521+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:42.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:42.547 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:42.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:42.540+0000 7fc165d9b640 0 -- 172.21.15.37:0/3035296749 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7fc1682204f0 msgr2=0x7fc1682228e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T04:01:42.573 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:42.599 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:42.600 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:42.600 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:42.600 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:42.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:01:42.628 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:42.626+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:42.725 INFO:tasks.rados.rados.0.smithi188.stdout:173: done (2 left) 2024-09-16T04:01:42.725 INFO:tasks.rados.rados.0.smithi188.stdout:174: done (1 left) 2024-09-16T04:01:42.725 INFO:tasks.rados.rados.0.smithi188.stdout:175: done (0 left) 2024-09-16T04:01:42.725 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 7 current snap is 13 2024-09-16T04:01:42.725 INFO:tasks.rados.rados.0.smithi188.stdout:176: seq_num 87 ranges {1556480=704512,2260992=688128,2949120=81920} 2024-09-16T04:01:42.729 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:42.726+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:42.761 INFO:tasks.rados.rados.0.smithi188.stdout:176: writing smithi18831615-7 from 1556480 to 2260992 tid 1 2024-09-16T04:01:42.774 INFO:tasks.rados.rados.0.smithi188.stdout:176: writing smithi18831615-7 from 2260992 to 2949120 tid 2 2024-09-16T04:01:42.775 INFO:tasks.rados.rados.0.smithi188.stdout:176: writing smithi18831615-7 from 2949120 to 3031040 tid 3 2024-09-16T04:01:42.776 INFO:tasks.rados.rados.0.smithi188.stdout:177: setattr oid 48 current snap is 13 2024-09-16T04:01:42.781 INFO:tasks.rados.rados.0.smithi188.stdout:178: snap_create 2024-09-16T04:01:42.784 INFO:tasks.rados.rados.0.smithi188.stdout:176: finishing write tid 1 to smithi18831615-7 2024-09-16T04:01:42.785 INFO:tasks.rados.rados.0.smithi188.stdout:176: finishing write tid 2 to smithi18831615-7 2024-09-16T04:01:42.786 INFO:tasks.rados.rados.0.smithi188.stdout:176: finishing write tid 3 to smithi18831615-7 2024-09-16T04:01:42.787 INFO:tasks.rados.rados.0.smithi188.stdout:176: finishing write tid 4 to smithi18831615-7 2024-09-16T04:01:42.788 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:42.788 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:42.792 INFO:tasks.rados.rados.0.smithi188.stdout:176: finishing write tid 5 to smithi18831615-7 2024-09-16T04:01:42.792 INFO:tasks.rados.rados.0.smithi188.stdout:176: oid 7 updating version 0 to 30 2024-09-16T04:01:42.792 INFO:tasks.rados.rados.0.smithi188.stdout:176: oid 7 updating version 30 to 31 2024-09-16T04:01:42.792 INFO:tasks.rados.rados.0.smithi188.stdout:176: oid 7 version 31 is already newer than 29 2024-09-16T04:01:42.792 INFO:tasks.rados.rados.0.smithi188.stdout:176: oid 7 updating version 31 to 32 2024-09-16T04:01:42.792 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 7 v 32 (ObjNum 87 snap 13 seq_num 87) dirty exists 2024-09-16T04:01:42.792 INFO:tasks.rados.rados.0.smithi188.stdout:176: left oid 7 (ObjNum 87 snap 13 seq_num 87) 2024-09-16T04:01:42.796 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 48 v 112 (ObjNum 83 snap 10 seq_num 83) dirty exists 2024-09-16T04:01:42.805 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:01:42.806 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:01:42.830 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:42.830+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:42.925 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:42.926 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:42.931 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:42.926+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:42.940 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:01:43.033 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:43.025+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:43.060 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:43.060 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:43.075 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:01:43.075 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:01:43.075 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T04:01:43.133 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:43.130+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:43.234 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:43.228+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:43.243 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:43.243 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:43.258 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:01:43.335 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:43.330+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:43.390 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:43.391 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:43.405 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:01:43.436 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:43.432+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:43.537 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:43.534+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:43.571 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:43.571 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:43.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:01:43.638 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:43.638+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:43.702 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:43.702 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:43.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:01:43.720 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-09-16T04:01:43.721 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T04:01:43.725 INFO:tasks.rados.rados.0.smithi188.stdout:176: done (2 left) 2024-09-16T04:01:43.725 INFO:tasks.rados.rados.0.smithi188.stdout:177: done (1 left) 2024-09-16T04:01:43.726 INFO:tasks.rados.rados.0.smithi188.stdout:178: done (0 left) 2024-09-16T04:01:43.726 INFO:tasks.rados.rados.0.smithi188.stdout:179: snap_create 2024-09-16T04:01:43.739 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:43.732+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:43.747 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T04:01:43.836 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:43.837 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:43.841 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:43.838+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:43.852 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:01:43.943 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:43.942+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:44.010 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:44.010 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:44.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:01:44.046 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:44.041+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:44.132 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:44.132 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:44.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:01:44.146 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:44.141+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:44.247 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:44.242+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:44.253 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:44.253 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:44.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:01:44.348 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:44.341+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:44.428 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:44.428 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:44.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:01:44.449 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:44.444+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:44.551 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:44.550+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:44.559 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:44.560 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:44.573 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:01:44.623 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:01:44.652 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:44.649+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:44.691 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:44.691 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:44.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:01:44.723 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T04:01:44.729 INFO:tasks.rados.rados.0.smithi188.stdout:179: done (0 left) 2024-09-16T04:01:44.729 INFO:tasks.rados.rados.0.smithi188.stdout:180: setattr oid 3 current snap is 15 2024-09-16T04:01:44.733 INFO:tasks.rados.rados.0.smithi188.stdout:181: read oid 45 snap -1 2024-09-16T04:01:44.733 INFO:tasks.rados.rados.0.smithi188.stdout:181: expect (ObjNum 86 snap 12 seq_num 86) 2024-09-16T04:01:44.733 INFO:tasks.rados.rados.0.smithi188.stdout:182: snap_create 2024-09-16T04:01:44.742 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 3 v 58 (ObjNum -388975728 snap 22024 seq_num 304) dirty exists 2024-09-16T04:01:44.751 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T04:01:44.753 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:44.750+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:44.754 INFO:tasks.rados.rados.0.smithi188.stdout:181: expect (ObjNum 86 snap 12 seq_num 86) 2024-09-16T04:01:44.784 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:44.784 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:44.805 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:44.827 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:44.827 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:44.828 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:44.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:01:44.848 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:44.849 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:44.853 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:44.848+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:44.866 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:44.866 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:44.891 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:44.892 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:44.909 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:44.931 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:44.948 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:44.948 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:44.954 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:44.954+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:44.969 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:44.969 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:44.989 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:44.990 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:45.006 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:45.006 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:45.012 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:45.012 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:45.020 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:01:45.028 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:45.029 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:45.050 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:45.050 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:45.066 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:45.066+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:45.073 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:45.064+0000 7faff3fff640 0 -- 172.21.15.37:0/356815807 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7faff4226590 msgr2=0x7faff4228a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:45.154 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:45.154 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:45.154 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:01:45.154 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:45.157 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:45.154+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:45.169 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:01:45.258 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:45.258+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:45.297 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:45.297 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:45.333 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:45.334 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:45.334 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:01:45.334 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:01:45.334 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:45.348 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:01:45.359 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:45.357+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:45.460 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:45.458+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:45.513 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4371.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:07.735249+0000", 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.776681332000003, 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074231499999999999, 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.514 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4371", 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.37:0/2503621667", 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735249+0000", 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735249+0000", 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735251+0000", 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6599999999999999e-06 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735265+0000", 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3862e-05 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735297+0000", 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1516000000000001e-05 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735313+0000", 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.666e-05 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735477+0000", 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016360100000000001 2024-09-16T04:01:45.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735495+0000", 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8416000000000001e-05 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.739219+0000", 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037237609999999999 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.739432+0000", 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021331 2024-09-16T04:01:45.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 12", 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.739697+0000", 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000265104 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.741578+0000", 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001880844 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:45.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.742529+0000", 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000950661 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.742607+0000", 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8196999999999997e-05 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.742672+0000", 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.520 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4371.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:08.740538+0000", 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.771392233999997, 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073770179999999999, 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4371", 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.37:0/2503621667", 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7 2024-09-16T04:01:45.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740538+0000", 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740538+0000", 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740541+0000", 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0469999999999998e-06 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740555+0000", 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4195999999999999e-05 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740586+0000", 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1195000000000002e-05 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740601+0000", 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5330000000000001e-05 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740778+0000", 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017643100000000001 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740795+0000", 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7102e-05 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.744656+0000", 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038614270000000002 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.744835+0000", 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017891000000000001 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 12", 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.745081+0000", 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024605599999999998 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.746669+0000", 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001587209 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.747763+0000", 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010943859999999999 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.747847+0000", 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4481000000000007e-05 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.747915+0000", 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7248000000000004e-05 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s2 40/33 ECSubWrite(tid=191, reqid=client.4646.0:318, at_version=40'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.028860+0000", 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.483069373999999, 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073691670000000003, 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.527 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028860+0000", 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028860+0000", 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028861+0000", 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8700000000000005e-07 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029461+0000", 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060003400000000005 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029496+0000", 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4965999999999997e-05 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029523+0000", 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7212e-05 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029703+0000", 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017971600000000001 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029726+0000", 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2816999999999999e-05 2024-09-16T04:01:45.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.036188+0000", 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064616860000000003 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.036230+0000", 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1949000000000001e-05 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s2 40/33 ECSubWrite(tid=192, reqid=client.4646.0:319, at_version=40'14, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.031462+0000", 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.480468077000001, 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048293019999999997, 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031462+0000", 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031462+0000", 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031464+0000", 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0229999999999999e-06 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031803+0000", 2024-09-16T04:01:45.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00033965999999999998 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031843+0000", 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9855999999999999e-05 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031867+0000", 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3258999999999999e-05 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031977+0000", 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110481 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031985+0000", 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7489999999999998e-06 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.036239+0000", 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042541219999999999 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.036291+0000", 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2151999999999998e-05 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s2 40/33 ECSubWrite(tid=193, reqid=client.4646.0:320, at_version=40'15, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.033231+0000", 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.478698769000001, 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042329289999999999, 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.536 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.033231+0000", 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.033231+0000", 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.033233+0000", 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8199999999999999e-06 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.033451+0000", 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021797200000000001 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.033482+0000", 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0822999999999999e-05 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.033503+0000", 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0922999999999999e-05 2024-09-16T04:01:45.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.033647+0000", 2024-09-16T04:01:45.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014449800000000001 2024-09-16T04:01:45.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.033664+0000", 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7159e-05 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.037413+0000", 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037484409999999999 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.037464+0000", 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1292999999999998e-05 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.540 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:349 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:29.923486+0000", 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.588443436, 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081109709999999998, 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 349 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923486+0000", 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923486+0000", 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923489+0000", 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6309999999999999e-06 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923502+0000", 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3203e-05 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923527+0000", 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4270999999999999e-05 2024-09-16T04:01:45.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923538+0000", 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1623999999999999e-05 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923713+0000", 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017450699999999999 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923728+0000", 2024-09-16T04:01:45.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5383000000000002e-05 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.926449+0000", 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002720779 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.926529+0000", 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9796000000000006e-05 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.931597+0000", 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005068777 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:350 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.588363785, 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092273240000000003, 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 350 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923567+0000", 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7e-07 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923571+0000", 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2830000000000003e-06 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923578+0000", 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2810000000000003e-06 2024-09-16T04:01:45.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923582+0000", 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1500000000000001e-06 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923849+0000", 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000266464 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:45.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923854+0000", 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.989e-06 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927191+0000", 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033374810000000002 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927250+0000", 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8369000000000003e-05 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.932793+0000", 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055438370000000002 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:351 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.588281531, 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097430910000000006, 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.551 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 351 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923649+0000", 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2399999999999999e-07 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923652+0000", 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.827e-06 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923659+0000", 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6420000000000001e-06 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923663+0000", 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7900000000000001e-06 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923914+0000", 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000251387 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923926+0000", 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1933000000000001e-05 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927543+0000", 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036171580000000001 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927607+0000", 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3925999999999997e-05 2024-09-16T04:01:45.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.933391+0000", 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005784104 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 51/33 ECSubWrite(tid=75, reqid=client.4646.0:384, at_version=51'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.6594115709999997, 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099262940000000004, 2024-09-16T04:01:45.556 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852519+0000", 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000004e-07 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853510+0000", 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099128100000000011 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853546+0000", 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5522000000000003e-05 2024-09-16T04:01:45.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853572+0000", 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5924000000000001e-05 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853747+0000", 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017537299999999999 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853774+0000", 2024-09-16T04:01:45.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7129000000000001e-05 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.862371+0000", 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085965390000000003 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.862445+0000", 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3942000000000005e-05 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.560 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 51/33 ECSubWrite(tid=76, reqid=client.4646.0:385, at_version=51'7, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.857984+0000", 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.6539453759999994, 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045052119999999998, 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.857984+0000", 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.857984+0000", 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.857985+0000", 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.001e-06 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.858128+0000", 2024-09-16T04:01:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000142213 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.858143+0000", 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5469e-05 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.858153+0000", 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0331e-05 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.858299+0000", 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000145613 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.858341+0000", 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.159e-05 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.862452+0000", 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041108719999999998 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.862490+0000", 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8123000000000001e-05 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 51/33 ECSubWrite(tid=77, reqid=client.4646.0:386, at_version=51'8, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.859858+0000", 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.6520723430000004, 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004915666, 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.565 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.859858+0000", 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.859858+0000", 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.859858+0000", 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4900000000000005e-07 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.859957+0000", 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9244000000000001e-05 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.859971+0000", 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3597999999999999e-05 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.859981+0000", 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7510000000000003e-06 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860106+0000", 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012559499999999999 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860132+0000", 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5483e-05 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.864709+0000", 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004577383 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.864773+0000", 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3862999999999995e-05 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 51/33 ECSubWrite(tid=78, reqid=client.4646.0:387, at_version=51'9, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.860068+0000", 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.6518619860000001, 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051979000000000001, 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860068+0000", 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860068+0000", 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860068+0000", 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860085+0000", 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6691999999999999e-05 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860096+0000", 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0421999999999999e-05 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860103+0000", 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5800000000000003e-06 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.861790+0000", 2024-09-16T04:01:45.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001687137 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.861820+0000", 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9269e-05 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.865210+0000", 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00339048 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.865266+0000", 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5739000000000003e-05 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 54/33 ECSubWrite(tid=33, reqid=client.4646.0:419, at_version=54'23, trim_to=0'0, roll_forward_to=42'21))", 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.796046+0000", 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.7158840709999996, 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040675019999999997, 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.574 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796046+0000", 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796046+0000", 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796047+0000", 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.297e-06 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796385+0000", 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00033762400000000002 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796423+0000", 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8078999999999999e-05 2024-09-16T04:01:45.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796447+0000", 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4507000000000001e-05 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796643+0000", 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019558299999999999 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796672+0000", 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9546000000000001e-05 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.800016+0000", 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033435969999999998 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.800113+0000", 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7269000000000007e-05 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.578 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 54/33 ECSubWrite(tid=34, reqid=client.4646.0:420, at_version=54'24, trim_to=0'0, roll_forward_to=42'21))", 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.809685+0000", 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.7022449789999996, 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052442230000000001, 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.809685+0000", 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.809685+0000", 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.809687+0000", 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.283e-06 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810192+0000", 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050483799999999997 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810225+0000", 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3383000000000002e-05 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810243+0000", 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7472000000000001e-05 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810434+0000", 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000191194 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810463+0000", 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8728999999999999e-05 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814856+0000", 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043929629999999997 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814929+0000", 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3361000000000005e-05 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 54/33 ECSubWrite(tid=211, reqid=osd.13.0:211, at_version=54'19, trim_to=0'0, roll_forward_to=53'17))", 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.932317+0000", 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.5796125950000004, 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050579199999999996, 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.583 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932317+0000", 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932317+0000", 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932318+0000", 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9000000000000002e-07 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932433+0000", 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011560799999999999 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932464+0000", 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0602999999999998e-05 2024-09-16T04:01:45.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932487+0000", 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3456e-05 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.933037+0000", 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054949000000000003 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.933065+0000", 2024-09-16T04:01:45.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8589999999999999e-05 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.937292+0000", 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042269109999999999 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.937375+0000", 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2872000000000002e-05 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpRead(3.ds1 56/33 ECSubRead(tid=38, to_read={3:b60aff58:::smithi18831615-34:head=0,180224,0}, subchunks={3:b60aff58:::smithi18831615-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.730293+0000", 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.7816366750000001, 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043211769999999998, 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730293+0000", 2024-09-16T04:01:45.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730293+0000", 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730295+0000", 2024-09-16T04:01:45.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.333e-06 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730331+0000", 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6196999999999999e-05 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730363+0000", 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2339000000000003e-05 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730390+0000", 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6962999999999999e-05 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730575+0000", 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018526 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.734614+0000", 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040390850000000004 2024-09-16T04:01:45.591 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 56/33 ECSubWrite(tid=93, reqid=osd.14.0:91, at_version=0'0, trim_to=0'0, roll_forward_to=56'15))", 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.735189+0000", 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.7767412650000001, 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051882819999999998, 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735189+0000", 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735189+0000", 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735190+0000", 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8500000000000001e-06 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735239+0000", 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8202999999999999e-05 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735264+0000", 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5412e-05 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735287+0000", 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2626999999999999e-05 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735440+0000", 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015364999999999999 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735466+0000", 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5786999999999999e-05 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.740251+0000", 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047844879999999999 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.740377+0000", 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000126265 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 56/33 ECSubWrite(tid=213, reqid=client.4646.0:430, at_version=56'21, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.757374+0000", 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.7545557220000001, 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051080140000000001, 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.596 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757374+0000", 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757374+0000", 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757375+0000", 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7700000000000004e-07 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758147+0000", 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077236600000000002 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758182+0000", 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5040000000000003e-05 2024-09-16T04:01:45.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758206+0000", 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3399000000000001e-05 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758401+0000", 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019587700000000001 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758434+0000", 2024-09-16T04:01:45.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2252000000000002e-05 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.762394+0000", 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039598250000000002 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.762482+0000", 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8578000000000005e-05 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 56/33 ECSubWrite(tid=214, reqid=osd.13.0:214, at_version=56'23, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.760746+0000", 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.7511835140000001, 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049967010000000001, 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.601 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760746+0000", 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760746+0000", 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760748+0000", 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.973e-06 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760801+0000", 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2432999999999998e-05 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760827+0000", 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6291999999999998e-05 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760842+0000", 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5177000000000001e-05 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760978+0000", 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013546700000000001 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761003+0000", 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5493000000000001e-05 2024-09-16T04:01:45.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.765628+0000", 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004624548 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.765743+0000", 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115318 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 56/33 ECSubWrite(tid=215, reqid=client.4646.0:431, at_version=56'24, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.763773+0000", 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.7481565880000001, 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045214269999999997, 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.763773+0000", 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.763773+0000", 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.763775+0000", 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0700000000000001e-06 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764206+0000", 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043073200000000002 2024-09-16T04:01:45.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764237+0000", 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1170000000000001e-05 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764253+0000", 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5305999999999999e-05 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764401+0000", 2024-09-16T04:01:45.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000148016 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764428+0000", 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7189999999999999e-05 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.768219+0000", 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037907990000000001 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:45.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.768295+0000", 2024-09-16T04:01:45.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6143999999999995e-05 2024-09-16T04:01:45.610 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.610 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.610 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.610 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:45.610 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:45.610 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:45.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:01:45.612 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:45.610+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:45.712 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:45.705+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:45.726 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T04:01:45.732 INFO:tasks.rados.rados.0.smithi188.stdout:180: done (2 left) 2024-09-16T04:01:45.732 INFO:tasks.rados.rados.0.smithi188.stdout:181: done (1 left) 2024-09-16T04:01:45.732 INFO:tasks.rados.rados.0.smithi188.stdout:182: done (0 left) 2024-09-16T04:01:45.732 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 8 current snap is 16 2024-09-16T04:01:45.732 INFO:tasks.rados.rados.0.smithi188.stdout:183: seq_num 88 ranges {131072=720896,851968=442368,1294336=327680} 2024-09-16T04:01:45.738 INFO:tasks.rados.rados.0.smithi188.stdout:183: writing smithi18831615-8 from 131072 to 851968 tid 1 2024-09-16T04:01:45.741 INFO:tasks.rados.rados.0.smithi188.stdout:183: writing smithi18831615-8 from 851968 to 1294336 tid 2 2024-09-16T04:01:45.744 INFO:tasks.rados.rados.0.smithi188.stdout:183: writing smithi18831615-8 from 1294336 to 1622016 tid 3 2024-09-16T04:01:45.744 INFO:tasks.rados.rados.0.smithi188.stdout:184: delete oid 50 current snap is 16 2024-09-16T04:01:45.749 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:01:45.754 INFO:tasks.rados.rados.0.smithi188.stdout:183: finishing write tid 1 to smithi18831615-8 2024-09-16T04:01:45.755 INFO:tasks.rados.rados.0.smithi188.stdout:183: finishing write tid 2 to smithi18831615-8 2024-09-16T04:01:45.757 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T04:01:45.759 INFO:tasks.rados.rados.0.smithi188.stdout:183: finishing write tid 3 to smithi18831615-8 2024-09-16T04:01:45.760 INFO:tasks.rados.rados.0.smithi188.stdout:183: finishing write tid 4 to smithi18831615-8 2024-09-16T04:01:45.762 INFO:tasks.rados.rados.0.smithi188.stdout:184: done (1 left) 2024-09-16T04:01:45.762 INFO:tasks.rados.rados.0.smithi188.stdout:185: read oid 11 snap -1 2024-09-16T04:01:45.762 INFO:tasks.rados.rados.0.smithi188.stdout:185: expect deleted 2024-09-16T04:01:45.762 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 27 current snap is 16 2024-09-16T04:01:45.762 INFO:tasks.rados.rados.0.smithi188.stdout:186: seq_num 90 ranges {1540096=753664,2293760=720896,3014656=49152} 2024-09-16T04:01:45.776 INFO:tasks.rados.rados.0.smithi188.stdout:186: writing smithi18831615-27 from 1540096 to 2293760 tid 1 2024-09-16T04:01:45.780 INFO:tasks.rados.rados.0.smithi188.stdout:186: writing smithi18831615-27 from 2293760 to 3014656 tid 2 2024-09-16T04:01:45.781 INFO:tasks.rados.rados.0.smithi188.stdout:186: writing smithi18831615-27 from 3014656 to 3063808 tid 3 2024-09-16T04:01:45.781 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 33 current snap is 16 2024-09-16T04:01:45.781 INFO:tasks.rados.rados.0.smithi188.stdout:187: seq_num 91 ranges {0=770048,770048=475136,1245184=294912} 2024-09-16T04:01:45.785 INFO:tasks.rados.rados.0.smithi188.stdout:187: writing smithi18831615-33 from 0 to 770048 tid 1 2024-09-16T04:01:45.788 INFO:tasks.rados.rados.0.smithi188.stdout:187: writing smithi18831615-33 from 770048 to 1245184 tid 2 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:187: writing smithi18831615-33 from 1245184 to 1540096 tid 3 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:188: rollback oid 24 current snap is 16 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 24 to 15 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:183: finishing write tid 5 to smithi18831615-8 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:183: oid 8 updating version 0 to 30 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:183: oid 8 version 30 is already newer than 28 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:183: oid 8 version 30 is already newer than 27 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:183: oid 8 version 30 is already newer than 29 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 8 v 30 (ObjNum 88 snap 16 seq_num 88) dirty exists 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:183: left oid 8 (ObjNum 88 snap 16 seq_num 88) 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:183: done (4 left) 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:185: done (3 left) 2024-09-16T04:01:45.791 INFO:tasks.rados.rados.0.smithi188.stdout:189: snap_create 2024-09-16T04:01:45.798 INFO:tasks.rados.rados.0.smithi188.stdout:188: finishing rollback tid 0 to smithi18831615-24 2024-09-16T04:01:45.798 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:01:45.799 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:45.799 INFO:tasks.rados.rados.0.smithi188.stdout:186: finishing write tid 1 to smithi18831615-27 2024-09-16T04:01:45.800 INFO:tasks.rados.rados.0.smithi188.stdout:187: finishing write tid 1 to smithi18831615-33 2024-09-16T04:01:45.802 INFO:tasks.rados.rados.0.smithi188.stdout:187: finishing write tid 2 to smithi18831615-33 2024-09-16T04:01:45.802 INFO:tasks.rados.rados.0.smithi188.stdout:188: finishing rollback tid 1 to smithi18831615-24 2024-09-16T04:01:45.802 INFO:tasks.rados.rados.0.smithi188.stdout:186: finishing write tid 2 to smithi18831615-27 2024-09-16T04:01:45.802 INFO:tasks.rados.rados.0.smithi188.stdout:188: finishing rollback tid 2 to smithi18831615-24 2024-09-16T04:01:45.802 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 24 v 116 (ObjNum 70 snap 3 seq_num 70) dirty exists 2024-09-16T04:01:45.802 INFO:tasks.rados.rados.0.smithi188.stdout:186: finishing write tid 3 to smithi18831615-27 2024-09-16T04:01:45.802 INFO:tasks.rados.rados.0.smithi188.stdout:186: finishing write tid 4 to smithi18831615-27 2024-09-16T04:01:45.805 INFO:tasks.rados.rados.0.smithi188.stdout:187: finishing write tid 3 to smithi18831615-33 2024-09-16T04:01:45.805 INFO:tasks.rados.rados.0.smithi188.stdout:187: finishing write tid 4 to smithi18831615-33 2024-09-16T04:01:45.809 INFO:tasks.rados.rados.0.smithi188.stdout:187: finishing write tid 5 to smithi18831615-33 2024-09-16T04:01:45.809 INFO:tasks.rados.rados.0.smithi188.stdout:187: oid 33 updating version 0 to 33 2024-09-16T04:01:45.810 INFO:tasks.rados.rados.0.smithi188.stdout:187: oid 33 updating version 33 to 36 2024-09-16T04:01:45.810 INFO:tasks.rados.rados.0.smithi188.stdout:187: oid 33 version 36 is already newer than 34 2024-09-16T04:01:45.810 INFO:tasks.rados.rados.0.smithi188.stdout:187: oid 33 version 36 is already newer than 35 2024-09-16T04:01:45.810 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 33 v 36 (ObjNum 91 snap 16 seq_num 91) dirty exists 2024-09-16T04:01:45.810 INFO:tasks.rados.rados.0.smithi188.stdout:187: left oid 33 (ObjNum 91 snap 16 seq_num 91) 2024-09-16T04:01:45.812 INFO:tasks.rados.rados.0.smithi188.stdout:186: finishing write tid 5 to smithi18831615-27 2024-09-16T04:01:45.812 INFO:tasks.rados.rados.0.smithi188.stdout:186: oid 27 updating version 0 to 11 2024-09-16T04:01:45.812 INFO:tasks.rados.rados.0.smithi188.stdout:186: oid 27 updating version 11 to 13 2024-09-16T04:01:45.812 INFO:tasks.rados.rados.0.smithi188.stdout:186: oid 27 version 13 is already newer than 12 2024-09-16T04:01:45.812 INFO:tasks.rados.rados.0.smithi188.stdout:186: oid 27 version 13 is already newer than 10 2024-09-16T04:01:45.812 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 27 v 13 (ObjNum 90 snap 16 seq_num 90) dirty exists 2024-09-16T04:01:45.812 INFO:tasks.rados.rados.0.smithi188.stdout:186: left oid 27 (ObjNum 90 snap 16 seq_num 90) 2024-09-16T04:01:45.812 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:45.805+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:45.816 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:01:45.913 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:45.910+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:45.935 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:45.935 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:45.935 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:01:45.935 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:01:45.935 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:45.949 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:01:46.034 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:46.034+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:46.112 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:46.112 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=23, reqid=osd.15.0:23, at_version=41'17, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.904709+0000", 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.206338800000001, 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054382349999999996, 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904709+0000", 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904709+0000", 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904710+0000", 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3e-07 2024-09-16T04:01:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904764+0000", 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3995000000000001e-05 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904795+0000", 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1593000000000001e-05 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.904818+0000", 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2133e-05 2024-09-16T04:01:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.905345+0000", 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000527171 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.905370+0000", 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5633999999999999e-05 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910074+0000", 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00470339 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910147+0000", 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3589000000000003e-05 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=41'19, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.907264+0000", 2024-09-16T04:01:46.117 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.203783979000001, 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060462780000000004, 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907264+0000", 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907264+0000", 2024-09-16T04:01:46.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907266+0000", 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8530000000000001e-06 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907317+0000", 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0806000000000001e-05 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907355+0000", 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8262000000000001e-05 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907380+0000", 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4776000000000001e-05 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907566+0000", 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018675300000000001 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907592+0000", 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5775999999999999e-05 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.913228+0000", 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0056355909999999997 2024-09-16T04:01:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.913310+0000", 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2460999999999995e-05 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 41/33 ECSubWrite(tid=71, reqid=osd.4.0:69, at_version=0'0, trim_to=0'0, roll_forward_to=41'20))", 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.907893+0000", 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.203155326000001, 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054840339999999996, 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907893+0000", 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907893+0000", 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907894+0000", 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.883e-06 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907922+0000", 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7702999999999999e-05 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907958+0000", 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5339000000000002e-05 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.907980+0000", 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.234e-05 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.908186+0000", 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020639500000000001 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.908211+0000", 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4666000000000001e-05 2024-09-16T04:01:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.913295+0000", 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050841799999999998 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.913377+0000", 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1527999999999996e-05 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s1 41/33 ECSubWrite(tid=198, reqid=osd.13.0:198, at_version=41'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.920987+0000", 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.19006087, 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055428689999999997, 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920987+0000", 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920987+0000", 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.920988+0000", 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.37e-07 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921042+0000", 2024-09-16T04:01:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4561999999999997e-05 2024-09-16T04:01:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921116+0000", 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3955000000000006e-05 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921139+0000", 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3108999999999998e-05 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921291+0000", 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015138 2024-09-16T04:01:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.921315+0000", 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3747000000000002e-05 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.926443+0000", 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051287240000000003 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.926530+0000", 2024-09-16T04:01:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6655000000000003e-05 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 51/33 ECSubWrite(tid=89, reqid=osd.4.0:89, at_version=51'33, trim_to=0'0, roll_forward_to=50'31))", 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.720927+0000", 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.3901207919999994, 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053388589999999996, 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.720927+0000", 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.720927+0000", 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.720928+0000", 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.63e-07 2024-09-16T04:01:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.720991+0000", 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3238999999999995e-05 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.721023+0000", 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2298999999999997e-05 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.721053+0000", 2024-09-16T04:01:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0009999999999999e-05 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.721232+0000", 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017905300000000001 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.721261+0000", 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8883000000000001e-05 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.726126+0000", 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048653180000000004 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.726266+0000", 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013959399999999999 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:419 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~671744] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:01:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.3185402860000002, 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019325796999999999, 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 419 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:01:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792510+0000", 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5359999999999999e-06 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793316+0000", 2024-09-16T04:01:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00080598599999999996 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793351+0000", 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4473e-05 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793368+0000", 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6943e-05 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793504+0000", 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013629000000000001 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793707+0000", 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020315399999999999 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796655+0000", 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002948301 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.800082+0000", 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003426999 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.811562+0000", 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011479896999999999 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.811833+0000", 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027121799999999999 2024-09-16T04:01:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:420 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~770048] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.805716+0000", 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.30533213, 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014453887, 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 420 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.805716+0000", 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.805716+0000", 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.805720+0000", 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0130000000000004e-06 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806805+0000", 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001085014 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806848+0000", 2024-09-16T04:01:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2972999999999997e-05 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806871+0000", 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3206000000000001e-05 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807049+0000", 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017784699999999999 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807342+0000", 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029323499999999998 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.809728+0000", 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023861239999999999 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814729+0000", 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050004710000000003 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.819977+0000", 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052487189999999998 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.820170+0000", 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019228499999999999 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:421 3.ds0 3:bab6d723:::smithi18831615-7:head [setxattr _header (56) in=7b] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.806948+0000", 2024-09-16T04:01:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.3041002390000003, 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014216368, 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 421 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806948+0000", 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806948+0000", 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806950+0000", 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.894e-06 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806960+0000", 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0756e-05 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806982+0000", 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1993000000000001e-05 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806999+0000", 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6835999999999999e-05 2024-09-16T04:01:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.811960+0000", 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004960402 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.812241+0000", 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028163899999999998 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.813544+0000", 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013022999999999999 2024-09-16T04:01:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.817454+0000", 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039106820000000004 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.820833+0000", 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033792119999999999 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.821164+0000", 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000330654 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:422 3.ds0 3:bab6d723:::smithi18831615-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.3040221570000003, 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02126656, 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 422 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807027+0000", 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3969999999999999e-06 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807034+0000", 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1459999999999999e-06 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807055+0000", 2024-09-16T04:01:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.09e-05 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807070+0000", 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4932999999999999e-05 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814329+0000", 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007259138 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814519+0000", 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000189435 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.822102+0000", 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075835199999999998 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.822353+0000", 2024-09-16T04:01:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025044999999999999 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.828292+0000", 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059396409999999998 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 54/33 ECSubWrite(tid=103, reqid=osd.4.0:102, at_version=0'0, trim_to=0'0, roll_forward_to=54'46))", 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.814360+0000", 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.2966875709999997, 2024-09-16T04:01:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063381929999999998, 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814360+0000", 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814360+0000", 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814362+0000", 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.81e-06 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814390+0000", 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7820000000000001e-05 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814419+0000", 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9108999999999999e-05 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814440+0000", 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1387e-05 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814583+0000", 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014247399999999999 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814606+0000", 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3445999999999999e-05 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.820618+0000", 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006012051 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.820699+0000", 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0096e-05 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:427 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3888507060000004, 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018266843000000001, 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 427 2024-09-16T04:01:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722200+0000", 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6929999999999999e-06 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722212+0000", 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2577e-05 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722245+0000", 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2162999999999997e-05 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722262+0000", 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6843000000000001e-05 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722406+0000", 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000144668 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722421+0000", 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5271999999999999e-05 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.728611+0000", 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006189193 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.728887+0000", 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027599500000000002 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.740464+0000", 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011577439 2024-09-16T04:01:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:428 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3887486940000002, 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019155492999999999, 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 428 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722300+0000", 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7999999999999993e-07 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722308+0000", 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8259999999999993e-06 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722350+0000", 2024-09-16T04:01:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1730000000000002e-05 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722359+0000", 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5289999999999997e-06 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722581+0000", 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000221623 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722605+0000", 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4116000000000001e-05 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730680+0000", 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080751769999999994 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730940+0000", 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000259382 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741455+0000", 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010515129999999999 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:429 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3886718289999997, 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024602057, 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 429 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:01:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722377+0000", 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722382+0000", 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7770000000000002e-06 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722392+0000", 2024-09-16T04:01:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0020000000000001e-05 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722399+0000", 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4000000000000003e-06 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722655+0000", 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025568500000000002 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722664+0000", 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4129999999999995e-06 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741688+0000", 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019023476000000001 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741991+0000", 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030357799999999999 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.746978+0000", 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049869179999999999 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpRead(3.ds0 56/33 ECSubRead(tid=38, to_read={3:b60aff58:::smithi18831615-34:head=0,180224,0}, subchunks={3:b60aff58:::smithi18831615-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.729728+0000", 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3813203830000003, 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068514489999999999, 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.729728+0000", 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.729728+0000", 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.729728+0000", 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.729728+0000", 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.729828+0000", 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100658 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.729856+0000", 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7886e-05 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733547+0000", 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003691368 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.736579+0000", 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030315369999999999 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpRead(3.ds0 56/33 ECSubRead(tid=39, to_read={3:b60aff58:::smithi18831615-34:head=0,180224,0}, subchunks={3:b60aff58:::smithi18831615-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.731521+0000", 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3795272919999997, 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055018560000000003, 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.731521+0000", 2024-09-16T04:01:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.731521+0000", 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.731521+0000", 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.731521+0000", 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.731582+0000", 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0883e-05 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.731607+0000", 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5159000000000001e-05 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.736761+0000", 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051543029999999998 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.737022+0000", 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026151100000000002 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 56/33 ECSubWrite(tid=106, reqid=osd.4.0:106, at_version=56'49, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:01:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.733446+0000", 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3776018629999998, 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065202919999999996, 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733446+0000", 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733446+0000", 2024-09-16T04:01:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733447+0000", 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.37e-07 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733527+0000", 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0282000000000001e-05 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733551+0000", 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4165999999999999e-05 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733572+0000", 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0429999999999999e-05 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.734853+0000", 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001281487 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.734880+0000", 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6840000000000001e-05 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.739880+0000", 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049996750000000003 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.739966+0000", 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6774999999999995e-05 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 56/33 ECSubWrite(tid=107, reqid=osd.4.0:107, at_version=56'51, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.733637+0000", 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3774107889999998, 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087424419999999996, 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733637+0000", 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733637+0000", 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733638+0000", 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.11e-06 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733680+0000", 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1739000000000001e-05 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733746+0000", 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5726000000000001e-05 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.733766+0000", 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0653e-05 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735714+0000", 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019478080000000001 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735747+0000", 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3179999999999997e-05 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.742271+0000", 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065231780000000001 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.742380+0000", 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000109048 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s1 56/33 ECSubWrite(tid=213, reqid=client.4646.0:430, at_version=56'21, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.757813+0000", 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3532344910000003, 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082254649999999995, 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757813+0000", 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757813+0000", 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757815+0000", 2024-09-16T04:01:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.08e-06 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758111+0000", 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029650500000000002 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758135+0000", 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3693000000000002e-05 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758150+0000", 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4836e-05 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758291+0000", 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000141481 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758337+0000", 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6019000000000001e-05 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.765955+0000", 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076181679999999998 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.766039+0000", 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3683000000000005e-05 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s1 56/33 ECSubWrite(tid=214, reqid=osd.13.0:214, at_version=56'23, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.760983+0000", 2024-09-16T04:01:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3500648670000004, 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074326119999999999, 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760983+0000", 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760983+0000", 2024-09-16T04:01:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760985+0000", 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.103e-06 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761037+0000", 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2105999999999999e-05 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761076+0000", 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8383000000000002e-05 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761098+0000", 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2143000000000001e-05 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761293+0000", 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019490600000000001 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761312+0000", 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9199999999999999e-05 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.768164+0000", 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068518920000000001 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.768416+0000", 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025187899999999999 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s1 56/33 ECSubWrite(tid=215, reqid=client.4646.0:431, at_version=56'24, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.764226+0000", 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3468215140000002, 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055210989999999998, 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764226+0000", 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764226+0000", 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764229+0000", 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2560000000000001e-06 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764441+0000", 2024-09-16T04:01:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021199599999999999 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764504+0000", 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3188999999999994e-05 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764527+0000", 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2946999999999999e-05 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764685+0000", 2024-09-16T04:01:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015839399999999999 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.764713+0000", 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7325999999999999e-05 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.769673+0000", 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049603980000000004 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.769748+0000", 2024-09-16T04:01:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4592999999999995e-05 2024-09-16T04:01:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.207 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:46.209 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:01:46.209 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:46.206+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:46.309 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:46.304+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:46.377 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:46.377 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:46.377 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:01:46.377 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:46.391 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:01:46.410 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:46.408+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:46.509 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:46.509 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:46.509 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:01:46.509 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:01:46.510 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:46.513 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:46.508+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:46.523 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:01:46.615 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:46.610+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:46.648 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:46.648 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:01:46.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:01:46.648 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:01:46.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:297 3.cs0 3:3922c872:::smithi18831615-21:head [append 0~425984] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.899819+0000", 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.746631892, 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020147297000000002, 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 297 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.649 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.899819+0000", 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.899819+0000", 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.899822+0000", 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0029999999999999e-06 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900319+0000", 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049783800000000001 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900351+0000", 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1365999999999998e-05 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900368+0000", 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6840000000000001e-05 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900494+0000", 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012627999999999999 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.900509+0000", 2024-09-16T04:01:46.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5342999999999999e-05 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.904460+0000", 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039507700000000001 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.904668+0000", 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000208285 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.906930+0000", 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002261692 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.919087+0000", 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012156752999999999 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.919839+0000", 2024-09-16T04:01:46.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075222699999999995 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.919966+0000", 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012689999999999999 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:300 3.cs0 3:3922c872:::smithi18831615-21:head [setxattr _header (56) in=7b] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.911913+0000", 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.734537108000001, 2024-09-16T04:01:46.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019281432000000001, 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 300 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911913+0000", 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911913+0000", 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911914+0000", 2024-09-16T04:01:46.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4600000000000003e-07 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911920+0000", 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4500000000000003e-06 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911935+0000", 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5330000000000001e-05 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911945+0000", 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8530000000000004e-06 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.924549+0000", 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012604087 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.924819+0000", 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000269597 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.926100+0000", 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012815540000000001 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928427+0000", 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023271569999999998 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931023+0000", 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025953780000000002 2024-09-16T04:01:46.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931195+0000", 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017207999999999999 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:301 3.cs0 3:3922c872:::smithi18831615-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:01:46.661 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.734493163, 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024704912999999998, 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 301 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911958+0000", 2024-09-16T04:01:46.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911962+0000", 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.63e-06 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911976+0000", 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4027e-05 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911985+0000", 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7530000000000007e-06 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928563+0000", 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016578809 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928680+0000", 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117123 2024-09-16T04:01:46.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931348+0000", 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002667141 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931647+0000", 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029889700000000001 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.936662+0000", 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050156630000000001 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:365 3.cs0 3:389a65d8:::smithi18831615-2:head [read 0~32768,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.718790+0000", 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.9276601959999997, 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021834770999999999, 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.667 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 365 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718790+0000", 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718790+0000", 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718794+0000", 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0729999999999998e-06 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718810+0000", 2024-09-16T04:01:46.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5532e-05 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718844+0000", 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3949999999999999e-05 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718901+0000", 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7669e-05 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719047+0000", 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000145694 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719060+0000", 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.725018+0000", 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059579450000000001 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.725265+0000", 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024645500000000002 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.740625+0000", 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015360303 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:366 3.cs0 3:389a65d8:::smithi18831615-2:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.9275017660000007, 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022413889999999999, 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 366 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.673 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718950+0000", 2024-09-16T04:01:46.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.663e-06 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718960+0000", 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.893e-06 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718982+0000", 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1820000000000001e-05 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718999+0000", 2024-09-16T04:01:46.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7280000000000001e-05 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719156+0000", 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015617699999999999 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719166+0000", 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0832999999999999e-05 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.730920+0000", 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01175319 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.731179+0000", 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000259588 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.741363+0000", 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010183446000000001 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:367 3.cs0 3:389a65d8:::smithi18831615-2:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.9273963549999991, 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024825495, 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 367 2024-09-16T04:01:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719056+0000", 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4839999999999999e-06 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719063+0000", 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7959999999999996e-06 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719085+0000", 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1365999999999998e-05 2024-09-16T04:01:46.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719100+0000", 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4863e-05 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719209+0000", 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108978 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719218+0000", 2024-09-16T04:01:46.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1339999999999993e-06 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733259+0000", 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014041420000000001 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733873+0000", 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061368800000000004 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.743880+0000", 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010006766 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:368 3.fs0 3:fec5116f:::smithi18831615-8:head [read 0~131072,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.719146+0000", 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.9273047499999993, 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016688428000000002, 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.683 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 368 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719146+0000", 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719146+0000", 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719147+0000", 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719155+0000", 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8529999999999992e-06 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719175+0000", 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9986000000000001e-05 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719189+0000", 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4343e-05 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719268+0000", 2024-09-16T04:01:46.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9066000000000005e-05 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719278+0000", 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9000000000000001e-06 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.725227+0000", 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059491550000000002 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.725470+0000", 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024276199999999999 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.735834+0000", 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010363933000000001 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.688 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:369 3.fs0 3:fec5116f:::smithi18831615-8:head [read 0~131072,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.719218+0000", 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.9272326409999998, 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019962819, 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 369 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.689 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719218+0000", 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719218+0000", 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719219+0000", 2024-09-16T04:01:46.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3170000000000001e-06 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719227+0000", 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9459999999999998e-06 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719246+0000", 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8723e-05 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719260+0000", 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3973e-05 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719334+0000", 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4139000000000003e-05 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719342+0000", 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.763e-06 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.730135+0000", 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010793647999999999 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.730390+0000", 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025506900000000002 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.739181+0000", 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087902410000000007 2024-09-16T04:01:46.693 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:370 3.fs0 3:fec5116f:::smithi18831615-8:head [read 0~131072,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.719304+0000", 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.9271459399999991, 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022086333, 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.694 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 370 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719304+0000", 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719304+0000", 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719306+0000", 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.314e-06 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719313+0000", 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.396e-06 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719333+0000", 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9817e-05 2024-09-16T04:01:46.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719349+0000", 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6429000000000001e-05 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719415+0000", 2024-09-16T04:01:46.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5778999999999995e-05 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719423+0000", 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5970000000000003e-06 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.731971+0000", 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012548020999999999 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.732207+0000", 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000236596 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.741391+0000", 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091833839999999993 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.699 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:383 3.cs0 3:385a92e2:::smithi18831615-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.839875+0000", 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.8065751199999998, 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019389956999999999, 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.700 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 383 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.839875+0000", 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.839875+0000", 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.839876+0000", 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2999999999999999e-07 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.839881+0000", 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6600000000000003e-06 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.839892+0000", 2024-09-16T04:01:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1586999999999999e-05 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.839911+0000", 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8019000000000001e-05 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.851405+0000", 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011494674999999999 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.851544+0000", 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000139154 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.854660+0000", 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031159949999999999 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.855156+0000", 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049591700000000002 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.859265+0000", 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041090199999999997 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:393 3.cs0 3:35c140c5:::smithi18831615-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873305+0000", 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7731450730000002, 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022093642, 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 393 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873305+0000", 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873305+0000", 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873306+0000", 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1900000000000003e-07 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873308+0000", 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4080000000000001e-06 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873315+0000", 2024-09-16T04:01:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6560000000000003e-06 2024-09-16T04:01:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873319+0000", 2024-09-16T04:01:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2939999999999999e-06 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.880138+0000", 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068183810000000001 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.880209+0000", 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0953999999999999e-05 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.882434+0000", 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022255539999999998 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.882561+0000", 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012717100000000001 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895399+0000", 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012837705 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:394 3.cs0 3:3631f100:::smithi18831615-24:head [read 0~1572864,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873325+0000", 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7731251130000008, 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017725152000000001, 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 394 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873325+0000", 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873325+0000", 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873326+0000", 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1800000000000001e-07 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873329+0000", 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9670000000000001e-06 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873334+0000", 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3979999999999998e-06 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873339+0000", 2024-09-16T04:01:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5029999999999999e-06 2024-09-16T04:01:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.880260+0000", 2024-09-16T04:01:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0069217810000000001 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.880392+0000", 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000131595 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.891050+0000", 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01065849 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:395 3.cs0 3:3631f100:::smithi18831615-24:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873344+0000", 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7731067290000002, 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020341341999999998, 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 395 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873344+0000", 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873344+0000", 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873344+0000", 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6399999999999998e-07 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873347+0000", 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4679999999999999e-06 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873352+0000", 2024-09-16T04:01:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.412e-06 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873358+0000", 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2450000000000003e-06 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.881206+0000", 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078479899999999991 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.881333+0000", 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000127014 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.893685+0000", 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012351849 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:396 3.cs0 3:3631f100:::smithi18831615-24:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:01:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7730869069999997, 2024-09-16T04:01:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024842948, 2024-09-16T04:01:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 396 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.01e-07 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873368+0000", 2024-09-16T04:01:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.092e-06 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873374+0000", 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5949999999999998e-06 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873378+0000", 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0949999999999998e-06 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.886123+0000", 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012745645999999999 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.886245+0000", 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012146699999999999 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.898206+0000", 2024-09-16T04:01:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011961652 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:400 3.cs0 3:3ac353cc:::smithi18831615-37:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873473+0000", 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7729777260000006, 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021013074, 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 400 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873473+0000", 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873473+0000", 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873473+0000", 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1300000000000001e-07 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873475+0000", 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.88e-06 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873480+0000", 2024-09-16T04:01:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2249999999999999e-06 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873484+0000", 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0999999999999997e-06 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.886699+0000", 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013215127 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.886757+0000", 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7657e-05 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.887313+0000", 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00055572399999999995 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.891115+0000", 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003801891 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.894237+0000", 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031226650000000002 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.894486+0000", 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000248392 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:401 3.cs0 3:3ac353cc:::smithi18831615-37:head [rollback 7] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7729606360000005, 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02229078, 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 401 2024-09-16T04:01:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:01:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1799999999999995e-07 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873492+0000", 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.657e-06 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873499+0000", 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.131e-06 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873504+0000", 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.305e-06 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888149+0000", 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014644292999999999 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888207+0000", 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8229000000000001e-05 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888580+0000", 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037271 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.891242+0000", 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026624280000000001 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895627+0000", 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043851180000000003 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895781+0000", 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015339100000000001 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:402 3.cs0 3:3ac353cc:::smithi18831615-37:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7729404930000001, 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022474063999999998, 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 402 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0499999999999999e-07 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873531+0000", 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0361e-05 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873538+0000", 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2219999999999996e-06 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873543+0000", 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8189999999999999e-06 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889058+0000", 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015515467 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889129+0000", 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0946999999999997e-05 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889436+0000", 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030638799999999999 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.891244+0000", 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001808289 2024-09-16T04:01:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895920+0000", 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046766380000000003 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895984+0000", 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3528000000000006e-05 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:453 3.fs0 3:fec5116f:::smithi18831615-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:45.744976+0000", 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.90147416000000002, 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018609654999999999, 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 453 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744976+0000", 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744976+0000", 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744977+0000", 2024-09-16T04:01:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9100000000000004e-07 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744979+0000", 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.249e-06 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744985+0000", 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9200000000000001e-06 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744989+0000", 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0929999999999996e-06 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.757783+0000", 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012793577 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.757872+0000", 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9828999999999997e-05 2024-09-16T04:01:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.761403+0000", 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003530633 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.761526+0000", 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012251100000000001 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.763586+0000", 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020603520000000001 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:454 3.fs0 3:f669afff:::smithi18831615-50:head [delete] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:45.744995+0000", 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.90145568899999995, 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017225118000000001, 2024-09-16T04:01:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 454 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744995+0000", 2024-09-16T04:01:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744995+0000", 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744995+0000", 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2899999999999999e-07 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.744997+0000", 2024-09-16T04:01:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6819999999999999e-06 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.745002+0000", 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9250000000000003e-06 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.745007+0000", 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.938e-06 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.757903+0000", 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01289654 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.757962+0000", 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.842e-05 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.758560+0000", 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059824599999999998 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.760024+0000", 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001464311 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.762123+0000", 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020989799999999999 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.762220+0000", 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6747000000000001e-05 2024-09-16T04:01:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:467 3.fs0 3:f22e48d8:::smithi18831615-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:45.791423+0000", 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.85502748100000003, 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018497311999999998, 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 467 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.791423+0000", 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.791423+0000", 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.791424+0000", 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.791428+0000", 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6899999999999998e-06 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.791441+0000", 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3043e-05 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.791450+0000", 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3000000000000007e-06 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.794559+0000", 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031094959999999998 2024-09-16T04:01:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.794648+0000", 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8200000000000003e-05 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.806407+0000", 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011758953000000001 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.806465+0000", 2024-09-16T04:01:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8628999999999997e-05 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.809920+0000", 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034551310000000002 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:46.762 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:46.763 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-09-16T04:01:46.764 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T04:01:46.765 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:01:46.766 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:186: done (3 left) 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:187: done (2 left) 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:188: done (1 left) 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:189: done (0 left) 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 38 current snap is 17 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:190: seq_num 92 ranges {622592=786432,1409024=491520,1900544=278528} 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:190: writing smithi18831615-38 from 622592 to 1409024 tid 1 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:190: writing smithi18831615-38 from 1409024 to 1900544 tid 2 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:190: writing smithi18831615-38 from 1900544 to 2179072 tid 3 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:191: read oid 44 snap -1 2024-09-16T04:01:46.768 INFO:tasks.rados.rados.0.smithi188.stdout:191: expect (ObjNum 69 snap 2 seq_num 69) 2024-09-16T04:01:46.769 INFO:tasks.rados.rados.0.smithi188.stdout:192: snap_create 2024-09-16T04:01:46.769 INFO:tasks.rados.rados.0.smithi188.stdout:190: finishing write tid 1 to smithi18831615-38 2024-09-16T04:01:46.769 INFO:tasks.rados.rados.0.smithi188.stdout:190: finishing write tid 2 to smithi18831615-38 2024-09-16T04:01:46.769 INFO:tasks.rados.rados.0.smithi188.stdout:190: finishing write tid 3 to smithi18831615-38 2024-09-16T04:01:46.769 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:46.762+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:46.769 INFO:tasks.rados.rados.0.smithi188.stdout:191: expect (ObjNum 69 snap 2 seq_num 69) 2024-09-16T04:01:46.865 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:46.856+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:46.867 INFO:tasks.rados.rados.0.smithi188.stdout:190: finishing write tid 4 to smithi18831615-38 2024-09-16T04:01:46.867 INFO:tasks.rados.rados.0.smithi188.stdout:190: finishing write tid 5 to smithi18831615-38 2024-09-16T04:01:46.867 INFO:tasks.rados.rados.0.smithi188.stdout:190: oid 38 updating version 0 to 119 2024-09-16T04:01:46.867 INFO:tasks.rados.rados.0.smithi188.stdout:190: oid 38 updating version 119 to 120 2024-09-16T04:01:46.867 INFO:tasks.rados.rados.0.smithi188.stdout:190: oid 38 updating version 120 to 121 2024-09-16T04:01:46.868 INFO:tasks.rados.rados.0.smithi188.stdout:190: oid 38 version 121 is already newer than 118 2024-09-16T04:01:46.868 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 38 v 121 (ObjNum 92 snap 17 seq_num 92) dirty exists 2024-09-16T04:01:46.868 INFO:tasks.rados.rados.0.smithi188.stdout:190: left oid 38 (ObjNum 92 snap 17 seq_num 92) 2024-09-16T04:01:46.893 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:46.893 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:46.893 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:01:46.893 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:46.909 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:01:46.965 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:46.962+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:47.031 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:01:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:01:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:01:47.031 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:47.045 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:01:47.066 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:47.060+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:47.167 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:47.160+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:106 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~589824,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.932645+0000", 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.239852991999999, 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0062299809999999999, 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 106 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.932645+0000", 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.932645+0000", 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.932646+0000", 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.398e-06 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933166+0000", 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052027300000000002 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933189+0000", 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2221999999999998e-05 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933200+0000", 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1442e-05 2024-09-16T04:01:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933268+0000", 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7545999999999993e-05 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933413+0000", 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014544699999999999 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.934620+0000", 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012073520000000001 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936195+0000", 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001574443 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.938788+0000", 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025934930000000001 2024-09-16T04:01:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.938875+0000", 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6365000000000004e-05 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:107 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.933223+0000", 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.239274436999999, 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074324209999999998, 2024-09-16T04:01:47.182 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 107 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933223+0000", 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933223+0000", 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933224+0000", 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1650000000000001e-06 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933229+0000", 2024-09-16T04:01:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5249999999999999e-06 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933237+0000", 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7279999999999995e-06 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933245+0000", 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6319999999999997e-06 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.935333+0000", 2024-09-16T04:01:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020876430000000001 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.935414+0000", 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1515999999999997e-05 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.935785+0000", 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037085299999999998 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936626+0000", 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084103900000000002 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.940589+0000", 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039625709999999998 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.940656+0000", 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6748999999999994e-05 2024-09-16T04:01:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:108 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.239241942, 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014725059, 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 108 2024-09-16T04:01:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:01:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:01:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2399999999999998e-07 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933259+0000", 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.937e-06 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933267+0000", 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6229999999999997e-06 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933273+0000", 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936240+0000", 2024-09-16T04:01:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002967365 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936330+0000", 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0200999999999999e-05 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.940982+0000", 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046523210000000001 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.941058+0000", 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5798999999999997e-05 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.947981+0000", 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006922615 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:139 3.5s0 3:a37b04c5:::smithi18831615-47:head [append 0~786432,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.980591+0000", 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.191906615000001, 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066524640000000003, 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 139 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980591+0000", 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980591+0000", 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980593+0000", 2024-09-16T04:01:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.598e-06 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980971+0000", 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037804800000000002 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980988+0000", 2024-09-16T04:01:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7509000000000001e-05 2024-09-16T04:01:47.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.980997+0000", 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5660000000000007e-06 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981057+0000", 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9997000000000002e-05 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981176+0000", 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011877799999999999 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.982015+0000", 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00083982700000000002 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.984671+0000", 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026555139999999999 2024-09-16T04:01:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987160+0000", 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024886560000000001 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987244+0000", 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3971e-05 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:140 3.5s0 3:a37b04c5:::smithi18831615-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.981017+0000", 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.191480831, 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006474301, 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 140 2024-09-16T04:01:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981017+0000", 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981017+0000", 2024-09-16T04:01:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981018+0000", 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2799999999999996e-07 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981023+0000", 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4700000000000001e-06 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981031+0000", 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3569999999999998e-06 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981038+0000", 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2740000000000004e-06 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.982734+0000", 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016962220000000001 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.982816+0000", 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1743999999999995e-05 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.983075+0000", 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000259213 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.984697+0000", 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001622615 2024-09-16T04:01:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987396+0000", 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026981140000000002 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987491+0000", 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5663999999999998e-05 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:141 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.191450691, 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087077830000000002, 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 141 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981048+0000", 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981051+0000", 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1659999999999998e-06 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981057+0000", 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6229999999999999e-06 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981063+0000", 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6550000000000001e-06 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.983220+0000", 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021567890000000001 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.983291+0000", 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1326000000000001e-05 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987840+0000", 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045492309999999999 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987922+0000", 2024-09-16T04:01:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1205000000000006e-05 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.989755+0000", 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018333049999999999 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:171 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:22.028441+0000", 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.144056719999998, 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007731295, 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 171 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028441+0000", 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028441+0000", 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028442+0000", 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.37e-07 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028445+0000", 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8290000000000001e-06 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028454+0000", 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.371e-06 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028460+0000", 2024-09-16T04:01:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0440000000000001e-06 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.029650+0000", 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011897570000000001 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.029706+0000", 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6484000000000003e-05 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030075+0000", 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036891499999999999 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030774+0000", 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069939000000000004 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036074+0000", 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052994870000000003 2024-09-16T04:01:47.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036172+0000", 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8381000000000006e-05 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:172 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.144027745999999, 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091168310000000006, 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.219 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 172 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:01:47.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9900000000000001e-07 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028474+0000", 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6969999999999999e-06 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028482+0000", 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6420000000000007e-06 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028490+0000", 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.712e-06 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030305+0000", 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001815327 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030412+0000", 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107055 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036597+0000", 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061849360000000003 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036681+0000", 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4110999999999995e-05 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.037587+0000", 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090585199999999996 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.224 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:314 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~491520] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.161850576999999, 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013929950999999999, 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 314 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:01:47.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010651+0000", 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.467e-06 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011198+0000", 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054779799999999995 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011237+0000", 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8092000000000001e-05 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011254+0000", 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7232999999999999e-05 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011454+0000", 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020057300000000001 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011623+0000", 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016875899999999999 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014078+0000", 2024-09-16T04:01:47.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024548510000000001 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024181+0000", 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010102877 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024366+0000", 2024-09-16T04:01:47.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018482199999999999 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024577+0000", 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021147900000000001 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:315 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~770048] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.156252721000001, 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091517409999999997, 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 315 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:01:47.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016247+0000", 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1799999999999999e-06 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017297+0000", 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010498269999999999 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017327+0000", 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0133000000000001e-05 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017349+0000", 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017547+0000", 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019868199999999999 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017839+0000", 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029153000000000002 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020248+0000", 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024092649999999998 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024592+0000", 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043436849999999999 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025288+0000", 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069595499999999995 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025397+0000", 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010890799999999999 2024-09-16T04:01:47.236 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:316 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (56) in=7b] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.155084883000001, 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010644074999999999, 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 316 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017415+0000", 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.99e-06 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017424+0000", 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4129999999999995e-06 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017444+0000", 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0115999999999998e-05 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017463+0000", 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8199999999999999e-05 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022150+0000", 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046871079999999997 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022414+0000", 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000264491 2024-09-16T04:01:47.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023228+0000", 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081356500000000003 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025417+0000", 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021892959999999999 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.027816+0000", 2024-09-16T04:01:47.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023989509999999999 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028057+0000", 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000240945 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.242 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:317 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.154973809000001, 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017770083999999998, 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 317 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.243 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017526+0000", 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017533+0000", 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1960000000000002e-06 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017555+0000", 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2157000000000001e-05 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017574+0000", 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8298999999999999e-05 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024692+0000", 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071183920000000003 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024964+0000", 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027219700000000001 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029509+0000", 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045448850000000002 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031984+0000", 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024747929999999999 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.035294+0000", 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033101850000000002 2024-09-16T04:01:47.247 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 40/33 ECSubWriteReply(tid=18, last_complete=40'13, committed=1, applied=1))", 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017864+0000", 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.154633752999999, 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059170250000000002, 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017864+0000", 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017864+0000", 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017866+0000", 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.587e-06 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017874+0000", 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017911+0000", 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6869000000000002e-05 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017925+0000", 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3453e-05 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023759+0000", 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058348890000000002 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023781+0000", 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1600999999999999e-05 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.0s2 41/33 ECSubWrite(tid=58, reqid=client.4646.0:330, at_version=41'35, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:47.251 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.910330+0000", 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.262167893000001, 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086055409999999995, 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910330+0000", 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910330+0000", 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910331+0000", 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7699999999999992e-07 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910381+0000", 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0118999999999999e-05 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910409+0000", 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7959000000000001e-05 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.910429+0000", 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9930000000000001e-05 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911102+0000", 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000673228 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.911126+0000", 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3536999999999998e-05 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918848+0000", 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077223389999999999 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.918935+0000", 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7452000000000002e-05 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.0s2 41/33 ECSubWrite(tid=61, reqid=osd.14.0:60, at_version=0'0, trim_to=0'0, roll_forward_to=41'39))", 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.930471+0000", 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.242026543000001, 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060718669999999999, 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.256 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930471+0000", 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930471+0000", 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930473+0000", 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.829e-06 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930525+0000", 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1693000000000001e-05 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930559+0000", 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4402e-05 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.930596+0000", 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6896e-05 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931008+0000", 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041160799999999999 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.931032+0000", 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4488999999999999e-05 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.936440+0000", 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054074300000000004 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.936543+0000", 2024-09-16T04:01:47.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010352 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.0s2 51/33 ECSubWrite(tid=73, reqid=client.4646.0:373, at_version=51'51, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.733567+0000", 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.43893081, 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061863739999999997, 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:47.261 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733567+0000", 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733567+0000", 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733568+0000", 2024-09-16T04:01:47.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5570000000000001e-06 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733615+0000", 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6428999999999999e-05 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733648+0000", 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3225999999999997e-05 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733671+0000", 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2665999999999999e-05 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.734051+0000", 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038053300000000001 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.734077+0000", 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5188999999999999e-05 2024-09-16T04:01:47.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.739668+0000", 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055910329999999996 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.739753+0000", 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5741000000000005e-05 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s4 51/33 ECSubWrite(tid=75, reqid=client.4646.0:384, at_version=51'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.852557+0000", 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.31994042, 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087782580000000006, 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852557+0000", 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852557+0000", 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852560+0000", 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8200000000000001e-06 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853239+0000", 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067888699999999998 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853289+0000", 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0346000000000002e-05 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853317+0000", 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7203e-05 2024-09-16T04:01:47.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853503+0000", 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018615600000000001 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853530+0000", 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6896000000000001e-05 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.861245+0000", 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077158360000000002 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.861336+0000", 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0113999999999998e-05 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.270 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s3 56/33 ECSubWrite(tid=213, reqid=client.4646.0:430, at_version=56'21, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.758019+0000", 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.4144785779999998, 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081202689999999994, 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758019+0000", 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758019+0000", 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758020+0000", 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1629999999999999e-06 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758875+0000", 2024-09-16T04:01:47.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00085437800000000004 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758970+0000", 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5694999999999998e-05 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758994+0000", 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3353000000000002e-05 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.759426+0000", 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000432476 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.759454+0000", 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7783000000000001e-05 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.766078+0000", 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066242790000000003 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.766139+0000", 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1142000000000005e-05 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s3 56/33 ECSubWrite(tid=214, reqid=osd.13.0:214, at_version=56'23, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.760982+0000", 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.4115158599999997, 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065009869999999997, 2024-09-16T04:01:47.275 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760982+0000", 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760982+0000", 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.760984+0000", 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.68e-06 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761025+0000", 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1158999999999997e-05 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761048+0000", 2024-09-16T04:01:47.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3176e-05 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.761064+0000", 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5872999999999999e-05 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.762721+0000", 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001657724 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.762747+0000", 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5097e-05 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.767411+0000", 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046647060000000002 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.767483+0000", 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1571999999999998e-05 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.279 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.8s5 60/33 ECSubWrite(tid=12, reqid=client.4646.0:458, at_version=60'10, trim_to=0'0, roll_forward_to=41'8))", 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:45.786346+0000", 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.386151965, 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057742879999999998, 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:01:47.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.786346+0000", 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.786346+0000", 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.786346+0000", 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.03e-07 2024-09-16T04:01:47.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.786961+0000", 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061466500000000002 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.786976+0000", 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4769999999999999e-05 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:01:47.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.786986+0000", 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0373999999999999e-05 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.787163+0000", 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017660899999999999 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.787191+0000", 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8807e-05 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.792061+0000", 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048692010000000001 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:45.792120+0000", 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9459000000000001e-05 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:01:47.284 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:01:47.286 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:01:47.287 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:47.282+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:47.326 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:47.387 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:47.386+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:47.418 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:47.418 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:47.419 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:47.419 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:47.435 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:01:47.488 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:47.486+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:47.510 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:47.532 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:47.532 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:47.554 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:47.554 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:47.569 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:47.569 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:47.576 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:47.576 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:47.584 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:01:47.589 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:47.585+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:47.594 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:47.595 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:47.616 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:47.616 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:47.637 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:47.637 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:47.660 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:47.660 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:47.678 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:47.678 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:47.690 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:47.685+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:47.700 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:47.706 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:47.706 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T04:01:47.721 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:01:47.722 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:47.722 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:47.729 INFO:tasks.rados.rados.0.smithi188.stdout:190: done (2 left) 2024-09-16T04:01:47.729 INFO:tasks.rados.rados.0.smithi188.stdout:191: done (1 left) 2024-09-16T04:01:47.729 INFO:tasks.rados.rados.0.smithi188.stdout:192: done (0 left) 2024-09-16T04:01:47.729 INFO:tasks.rados.rados.0.smithi188.stdout:193: read oid 24 snap 13 2024-09-16T04:01:47.729 INFO:tasks.rados.rados.0.smithi188.stdout:193: expect (ObjNum 70 snap 3 seq_num 70) 2024-09-16T04:01:47.729 INFO:tasks.rados.rados.0.smithi188.stdout:194: delete oid 14 current snap is 18 2024-09-16T04:01:47.741 INFO:tasks.rados.rados.0.smithi188.stdout:194: done (1 left) 2024-09-16T04:01:47.742 INFO:tasks.rados.rados.0.smithi188.stdout:195: snap_create 2024-09-16T04:01:47.750 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T04:01:47.750 INFO:tasks.rados.rados.0.smithi188.stdout:193: expect (ObjNum 70 snap 3 seq_num 70) 2024-09-16T04:01:47.755 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:47.755 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:47.773 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:47.773 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:47.790 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:47.784+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:47.793 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:47.794 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:47.815 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:47.815 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:47.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:47.808+0000 7f6dc3fff640 0 -- 172.21.15.37:0/1574690283 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7f6dc4224040 msgr2=0x7f6dc42264b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:47.836 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:47.837 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:47.845 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:47.845 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:47.863 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:01:47.892 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:47.890+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:47.993 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:47.985+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:48.021 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:48.021 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:48.036 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:01:48.094 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:48.088+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:48.160 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:48.160 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:48.175 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:01:48.194 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:48.189+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:48.295 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:48.290+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:48.352 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:01:48.397 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:48.393+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:48.474 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:48.474 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:48.488 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:01:48.534 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:48.528+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:48.598 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:48.597+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:48.606 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:48.606 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:48.621 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:01:48.699 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:48.693+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:48.722 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T04:01:48.731 INFO:tasks.rados.rados.0.smithi188.stdout:193: done (1 left) 2024-09-16T04:01:48.731 INFO:tasks.rados.rados.0.smithi188.stdout:195: done (0 left) 2024-09-16T04:01:48.731 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 38 current snap is 19 2024-09-16T04:01:48.731 INFO:tasks.rados.rados.0.smithi188.stdout:196: seq_num 94 ranges {2179072=425984,2605056=786432,3391488=376832} 2024-09-16T04:01:48.744 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T04:01:48.768 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:01:48.773 INFO:tasks.rados.rados.0.smithi188.stdout:196: writing smithi18831615-38 from 2179072 to 2605056 tid 1 2024-09-16T04:01:48.785 INFO:tasks.rados.rados.0.smithi188.stdout:196: writing smithi18831615-38 from 2605056 to 3391488 tid 2 2024-09-16T04:01:48.791 INFO:tasks.rados.rados.0.smithi188.stdout:196: writing smithi18831615-38 from 3391488 to 3768320 tid 3 2024-09-16T04:01:48.791 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 32 current snap is 19 2024-09-16T04:01:48.791 INFO:tasks.rados.rados.0.smithi188.stdout:197: seq_num 95 ranges {524288=442368,966656=540672,1507328=622592} 2024-09-16T04:01:48.795 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:48.795 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:48.800 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:48.793+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:48.801 INFO:tasks.rados.rados.0.smithi188.stdout:197: writing smithi18831615-32 from 524288 to 966656 tid 1 2024-09-16T04:01:48.806 INFO:tasks.rados.rados.0.smithi188.stdout:197: writing smithi18831615-32 from 966656 to 1507328 tid 2 2024-09-16T04:01:48.813 INFO:tasks.rados.rados.0.smithi188.stdout:197: writing smithi18831615-32 from 1507328 to 2129920 tid 3 2024-09-16T04:01:48.813 INFO:tasks.rados.rados.0.smithi188.stdout:198: copy_from oid 44 from oid 21 current snap is 19 2024-09-16T04:01:48.813 INFO:tasks.rados.rados.0.smithi188.stdout:196: finishing write tid 1 to smithi18831615-38 2024-09-16T04:01:48.813 INFO:tasks.rados.rados.0.smithi188.stdout:196: finishing write tid 2 to smithi18831615-38 2024-09-16T04:01:48.813 INFO:tasks.rados.rados.0.smithi188.stdout:196: finishing write tid 3 to smithi18831615-38 2024-09-16T04:01:48.813 INFO:tasks.rados.rados.0.smithi188.stdout:196: finishing write tid 4 to smithi18831615-38 2024-09-16T04:01:48.813 INFO:tasks.rados.rados.0.smithi188.stdout:196: finishing write tid 5 to smithi18831615-38 2024-09-16T04:01:48.813 INFO:tasks.rados.rados.0.smithi188.stdout:196: oid 38 updating version 0 to 125 2024-09-16T04:01:48.814 INFO:tasks.rados.rados.0.smithi188.stdout:196: oid 38 version 125 is already newer than 123 2024-09-16T04:01:48.814 INFO:tasks.rados.rados.0.smithi188.stdout:196: oid 38 updating version 125 to 126 2024-09-16T04:01:48.814 INFO:tasks.rados.rados.0.smithi188.stdout:196: oid 38 version 126 is already newer than 124 2024-09-16T04:01:48.814 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 38 v 126 (ObjNum 94 snap 19 seq_num 94) dirty exists 2024-09-16T04:01:48.814 INFO:tasks.rados.rados.0.smithi188.stdout:196: left oid 38 (ObjNum 94 snap 19 seq_num 94) 2024-09-16T04:01:48.814 INFO:tasks.rados.rados.0.smithi188.stdout:196: done (2 left) 2024-09-16T04:01:48.814 INFO:tasks.rados.rados.0.smithi188.stdout:199: delete oid 3 current snap is 19 2024-09-16T04:01:48.815 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:01:48.821 INFO:tasks.rados.rados.0.smithi188.stdout:197: finishing write tid 1 to smithi18831615-32 2024-09-16T04:01:48.822 INFO:tasks.rados.rados.0.smithi188.stdout:197: finishing write tid 2 to smithi18831615-32 2024-09-16T04:01:48.824 INFO:tasks.rados.rados.0.smithi188.stdout:197: finishing write tid 3 to smithi18831615-32 2024-09-16T04:01:48.824 INFO:tasks.rados.rados.0.smithi188.stdout:199: done (2 left) 2024-09-16T04:01:48.824 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 31 current snap is 19 2024-09-16T04:01:48.825 INFO:tasks.rados.rados.0.smithi188.stdout:197: finishing write tid 4 to smithi18831615-32 2024-09-16T04:01:48.825 INFO:tasks.rados.rados.0.smithi188.stdout:200: seq_num 97 ranges {0=475136,475136=442368,917504=442368,1359872=278528} 2024-09-16T04:01:48.829 INFO:tasks.rados.rados.0.smithi188.stdout:200: writing smithi18831615-31 from 0 to 475136 tid 1 2024-09-16T04:01:48.833 INFO:tasks.rados.rados.0.smithi188.stdout:200: writing smithi18831615-31 from 475136 to 917504 tid 2 2024-09-16T04:01:48.838 INFO:tasks.rados.rados.0.smithi188.stdout:200: writing smithi18831615-31 from 917504 to 1359872 tid 3 2024-09-16T04:01:48.842 INFO:tasks.rados.rados.0.smithi188.stdout:200: writing smithi18831615-31 from 1359872 to 1638400 tid 4 2024-09-16T04:01:48.842 INFO:tasks.rados.rados.0.smithi188.stdout:201: snap_create 2024-09-16T04:01:48.842 INFO:tasks.rados.rados.0.smithi188.stdout:197: finishing write tid 5 to smithi18831615-32 2024-09-16T04:01:48.842 INFO:tasks.rados.rados.0.smithi188.stdout:197: oid 32 updating version 0 to 19 2024-09-16T04:01:48.843 INFO:tasks.rados.rados.0.smithi188.stdout:197: oid 32 version 19 is already newer than 18 2024-09-16T04:01:48.843 INFO:tasks.rados.rados.0.smithi188.stdout:197: oid 32 version 19 is already newer than 17 2024-09-16T04:01:48.843 INFO:tasks.rados.rados.0.smithi188.stdout:197: oid 32 version 19 is already newer than 16 2024-09-16T04:01:48.843 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 32 v 19 (ObjNum 95 snap 19 seq_num 95) dirty exists 2024-09-16T04:01:48.843 INFO:tasks.rados.rados.0.smithi188.stdout:197: left oid 32 (ObjNum 95 snap 19 seq_num 95) 2024-09-16T04:01:48.843 INFO:tasks.rados.rados.0.smithi188.stdout:200: finishing write tid 1 to smithi18831615-31 2024-09-16T04:01:48.844 INFO:tasks.rados.rados.0.smithi188.stdout:198: finishing copy_from racing read to smithi18831615-44 2024-09-16T04:01:48.845 INFO:tasks.rados.rados.0.smithi188.stdout:200: finishing write tid 2 to smithi18831615-31 2024-09-16T04:01:48.846 INFO:tasks.rados.rados.0.smithi188.stdout:200: finishing write tid 3 to smithi18831615-31 2024-09-16T04:01:48.846 INFO:tasks.rados.rados.0.smithi188.stdout:198: finishing copy_from to smithi18831615-44 2024-09-16T04:01:48.846 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 44 v 50 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-09-16T04:01:48.847 INFO:tasks.rados.rados.0.smithi188.stdout:200: finishing write tid 4 to smithi18831615-31 2024-09-16T04:01:48.848 INFO:tasks.rados.rados.0.smithi188.stdout:200: finishing write tid 5 to smithi18831615-31 2024-09-16T04:01:48.850 INFO:tasks.rados.rados.0.smithi188.stdout:200: finishing write tid 6 to smithi18831615-31 2024-09-16T04:01:48.850 INFO:tasks.rados.rados.0.smithi188.stdout:200: oid 31 updating version 0 to 24 2024-09-16T04:01:48.850 INFO:tasks.rados.rados.0.smithi188.stdout:200: oid 31 updating version 24 to 25 2024-09-16T04:01:48.850 INFO:tasks.rados.rados.0.smithi188.stdout:200: oid 31 updating version 25 to 26 2024-09-16T04:01:48.850 INFO:tasks.rados.rados.0.smithi188.stdout:200: oid 31 updating version 26 to 27 2024-09-16T04:01:48.850 INFO:tasks.rados.rados.0.smithi188.stdout:200: oid 31 version 27 is already newer than 23 2024-09-16T04:01:48.850 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 31 v 27 (ObjNum 97 snap 19 seq_num 97) dirty exists 2024-09-16T04:01:48.850 INFO:tasks.rados.rados.0.smithi188.stdout:200: left oid 31 (ObjNum 97 snap 19 seq_num 97) 2024-09-16T04:01:48.901 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:48.897+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:48.973 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:48.973 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:48.985 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:01:49.002 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:48.998+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:49.095 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:49.095 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:49.103 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:49.096+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:49.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:01:49.204 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:49.202+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:49.243 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:49.244 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:49.258 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:01:49.306 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:49.301+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:49.383 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:49.383 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:49.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:01:49.407 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:49.404+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:49.507 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:49.500+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:49.524 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:49.524 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:49.539 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:01:49.609 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:49.600+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:49.667 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:49.667 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:49.682 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:01:49.710 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:49.704+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:49.810 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:49.804+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:49.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:01:49.843 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T04:01:49.843 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-09-16T04:01:49.850 INFO:tasks.rados.rados.0.smithi188.stdout:197: done (3 left) 2024-09-16T04:01:49.850 INFO:tasks.rados.rados.0.smithi188.stdout:198: done (2 left) 2024-09-16T04:01:49.850 INFO:tasks.rados.rados.0.smithi188.stdout:200: done (1 left) 2024-09-16T04:01:49.850 INFO:tasks.rados.rados.0.smithi188.stdout:201: done (0 left) 2024-09-16T04:01:49.850 INFO:tasks.rados.rados.0.smithi188.stdout:202: read oid 35 snap -1 2024-09-16T04:01:49.850 INFO:tasks.rados.rados.0.smithi188.stdout:202: expect (ObjNum 66 snap 1 seq_num 66) 2024-09-16T04:01:49.850 INFO:tasks.rados.rados.0.smithi188.stdout:203: snap_create 2024-09-16T04:01:49.857 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:01:49.871 INFO:tasks.rados.rados.0.smithi188.stdout:202: expect (ObjNum 66 snap 1 seq_num 66) 2024-09-16T04:01:49.882 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T04:01:49.911 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:49.905+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:49.949 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:49.949 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:49.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:01:50.012 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:50.010+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:50.040 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:50.040 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:50.059 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:50.083 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:50.083 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:50.113 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:50.109+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:50.113 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:50.114 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:50.132 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:50.132 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:50.138 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:50.138 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:50.153 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:50.153 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:50.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:01:50.175 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:50.175 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:50.214 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:50.210+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:50.229 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:50.229 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:50.250 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:50.250 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:50.270 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:50.270 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:50.272 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:50.272 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:50.272 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:50.272 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:50.272 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:50.287 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:01:50.289 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:50.289 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:50.304 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:50.305 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:50.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:50.300+0000 7f2769763640 0 -- 172.21.15.37:0/2971146561 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7f276421f730 msgr2=0x7f2764221b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:50.315 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:50.308+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:50.415 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 40/33 ECSubWrite(tid=66, reqid=client.4646.0:302, at_version=40'8, trim_to=0'0, roll_forward_to=37'6))", 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:25.919876+0000", 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 24.492900964, 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025883859999999998, 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.919876+0000", 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.919876+0000", 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.919877+0000", 2024-09-16T04:01:50.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920263+0000", 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00038648099999999999 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920308+0000", 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4641999999999997e-05 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920338+0000", 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0185999999999999e-05 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920466+0000", 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012813000000000001 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.920483+0000", 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7255e-05 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.922353+0000", 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001870086 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:25.922464+0000", 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000110943 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/33 ECSubWrite(tid=69, reqid=osd.4.0:69, at_version=41'20, trim_to=0'0, roll_forward_to=37'18))", 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.898677+0000", 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.514099544, 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0047119279999999998, 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898677+0000", 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898677+0000", 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898678+0000", 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898738+0000", 2024-09-16T04:01:50.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9784999999999998e-05 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898777+0000", 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8822000000000002e-05 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898800+0000", 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3025999999999999e-05 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.899215+0000", 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041527300000000002 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.899243+0000", 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7555e-05 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.903310+0000", 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040673369999999999 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.903389+0000", 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9396999999999999e-05 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 41/33 ECSubWrite(tid=68, reqid=osd.4.0:68, at_version=41'10, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.898853+0000", 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.51392366, 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0065209559999999996, 2024-09-16T04:01:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898853+0000", 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898853+0000", 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898854+0000", 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898903+0000", 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8668000000000002e-05 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898929+0000", 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6346000000000002e-05 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898947+0000", 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7442999999999999e-05 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.900087+0000", 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011405289999999999 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.900113+0000", 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5893000000000001e-05 2024-09-16T04:01:50.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.905274+0000", 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0051606530000000003 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.905374+0000", 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000100574 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 41/33 ECSubWrite(tid=70, reqid=osd.4.0:70, at_version=41'12, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.511174377, 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0095027129999999994, 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901604+0000", 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.933e-06 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901660+0000", 2024-09-16T04:01:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5979000000000001e-05 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901691+0000", 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0338999999999999e-05 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901716+0000", 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901925+0000", 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020894899999999999 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901950+0000", 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5303000000000002e-05 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.911017+0000", 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0090665969999999992 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.911105+0000", 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8153999999999997e-05 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/33 ECSubWrite(tid=71, reqid=osd.4.0:69, at_version=0'0, trim_to=0'0, roll_forward_to=41'20))", 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.907303+0000", 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.505473811000002, 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0047894519999999996, 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.434 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907303+0000", 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907303+0000", 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907304+0000", 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.052e-06 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907317+0000", 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3019e-05 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907337+0000", 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0140999999999999e-05 2024-09-16T04:01:50.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907348+0000", 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0805000000000001e-05 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907489+0000", 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014113799999999999 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907513+0000", 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3952e-05 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.911951+0000", 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044380519999999996 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.912093+0000", 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014129299999999999 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.438 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 41/33 ECSubWrite(tid=72, reqid=osd.4.0:70, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.915094+0000", 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.497682648000001, 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00234155, 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915094+0000", 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915094+0000", 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915095+0000", 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8899999999999998e-07 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915111+0000", 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5336999999999999e-05 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915124+0000", 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915133+0000", 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1439999999999992e-06 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915263+0000", 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012980199999999999 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915289+0000", 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5588999999999998e-05 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.917340+0000", 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0020515199999999998 2024-09-16T04:01:50.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.917436+0000", 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5429999999999999e-05 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.as3 41/33 ECSubWrite(tid=13, reqid=osd.3.0:13, at_version=41'8, trim_to=0'0, roll_forward_to=38'6))", 2024-09-16T04:01:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.915854+0000", 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.496922907999998, 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0019640619999999999, 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915854+0000", 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915854+0000", 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915855+0000", 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915906+0000", 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0900999999999999e-05 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915935+0000", 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9268000000000002e-05 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.915956+0000", 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0682e-05 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.916144+0000", 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000188331 2024-09-16T04:01:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.916167+0000", 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3076000000000001e-05 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.917728+0000", 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015610349999999999 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.917818+0000", 2024-09-16T04:01:50.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0029000000000001e-05 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpRead(3.as3 44/33 ECSubRead(tid=18, to_read={3:57198856:::smithi18831615-42:head=0,176128,0}, subchunks={3:57198856:::smithi18831615-42:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.927012+0000", 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.485764866, 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018692120000000001, 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.448 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927012+0000", 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927012+0000", 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927013+0000", 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.243e-06 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927028+0000", 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4962e-05 2024-09-16T04:01:50.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927203+0000", 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000174365 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927234+0000", 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1526000000000003e-05 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928588+0000", 2024-09-16T04:01:50.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013534319999999999 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928881+0000", 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000293684 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpRead(3.as3 44/33 ECSubRead(tid=19, to_read={3:57198856:::smithi18831615-42:head=0,176128,0}, subchunks={3:57198856:::smithi18831615-42:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.927344+0000", 2024-09-16T04:01:50.452 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.485432743, 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001919341, 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927344+0000", 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927344+0000", 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927345+0000", 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.316e-06 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927367+0000", 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1253000000000001e-05 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927399+0000", 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2019e-05 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927427+0000", 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8668999999999999e-05 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928991+0000", 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015638119999999999 2024-09-16T04:01:50.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929264+0000", 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000272272 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 44/33 ECSubWrite(tid=82, reqid=client.4646.0:352, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.928975+0000", 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.483802099999998, 2024-09-16T04:01:50.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045655690000000002, 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928975+0000", 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928975+0000", 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928976+0000", 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929046+0000", 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0736999999999997e-05 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929078+0000", 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1612999999999997e-05 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929101+0000", 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3509000000000001e-05 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929281+0000", 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017924099999999999 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929307+0000", 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6625999999999999e-05 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.933440+0000", 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041322499999999996 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.933540+0000", 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010087299999999999 2024-09-16T04:01:50.460 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 44/33 ECSubWrite(tid=83, reqid=client.4646.0:352, at_version=0'0, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.934988+0000", 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.477788624999999, 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025391960000000001, 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934988+0000", 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934988+0000", 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.934990+0000", 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.857e-06 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.935039+0000", 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9211e-05 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.935224+0000", 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000184305 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.935516+0000", 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029253100000000001 2024-09-16T04:01:50.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.935815+0000", 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000298404 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.935840+0000", 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5133000000000001e-05 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.937424+0000", 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015842810000000001 2024-09-16T04:01:50.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.937527+0000", 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010347400000000001 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 49/33 ECSubWrite(tid=84, reqid=osd.4.0:84, at_version=49'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:35.034108+0000", 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.378668806, 2024-09-16T04:01:50.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001832235, 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034108+0000", 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034108+0000", 2024-09-16T04:01:50.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034109+0000", 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034159+0000", 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9988999999999999e-05 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034181+0000", 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1858999999999999e-05 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034196+0000", 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5566999999999999e-05 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034342+0000", 2024-09-16T04:01:50.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000145649 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.034357+0000", 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5193e-05 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.035867+0000", 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001510246 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.035940+0000", 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2991999999999995e-05 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 50/33 ECSubWrite(tid=86, reqid=client.4646.0:362, at_version=50'30, trim_to=0'0, roll_forward_to=49'28))", 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:35.749356+0000", 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.663421353, 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021355129999999999, 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.470 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749356+0000", 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749356+0000", 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749356+0000", 2024-09-16T04:01:50.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0399999999999995e-07 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749417+0000", 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0435e-05 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749452+0000", 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5012000000000003e-05 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749477+0000", 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.495e-05 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749590+0000", 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000112983 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.749605+0000", 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5092999999999999e-05 2024-09-16T04:01:50.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.751410+0000", 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018051549999999999 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:35.751491+0000", 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1180999999999994e-05 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 51/33 ECSubWrite(tid=89, reqid=osd.4.0:89, at_version=51'33, trim_to=0'0, roll_forward_to=50'31))", 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.720301+0000", 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.692475567000001, 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0077503939999999999, 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720301+0000", 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720301+0000", 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720302+0000", 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.44e-07 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720375+0000", 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3127999999999995e-05 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720406+0000", 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0911999999999998e-05 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720428+0000", 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1776e-05 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720652+0000", 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022443100000000001 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720675+0000", 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.319e-05 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727971+0000", 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072956649999999998 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.728052+0000", 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0648000000000004e-05 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 51/33 ECSubWrite(tid=90, reqid=osd.4.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=51'33))", 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.734117+0000", 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.678659416, 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021154630000000002, 2024-09-16T04:01:50.479 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734117+0000", 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734117+0000", 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734119+0000", 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7540000000000001e-06 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734149+0000", 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9832000000000002e-05 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734189+0000", 2024-09-16T04:01:50.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9752999999999997e-05 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734212+0000", 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3623000000000001e-05 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734440+0000", 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022743000000000001 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.734464+0000", 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4473000000000001e-05 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736135+0000", 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001670737 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.736233+0000", 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7861000000000004e-05 2024-09-16T04:01:50.483 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 51/33 ECSubWrite(tid=91, reqid=client.4646.0:374, at_version=51'35, trim_to=0'0, roll_forward_to=51'33))", 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.810646+0000", 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.602130867, 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003944075, 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.484 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.810646+0000", 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.810646+0000", 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.810648+0000", 2024-09-16T04:01:50.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9369999999999998e-06 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811104+0000", 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00045560099999999998 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811136+0000", 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811154+0000", 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8122999999999999e-05 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811328+0000", 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017394900000000001 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.811360+0000", 2024-09-16T04:01:50.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2292999999999997e-05 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.814508+0000", 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031482390000000002 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.814590+0000", 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1734e-05 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 51/33 ECSubWrite(tid=93, reqid=client.4646.0:375, at_version=51'36, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.819443+0000", 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.593333787000001, 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002547821, 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819443+0000", 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819443+0000", 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819445+0000", 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6139999999999999e-06 2024-09-16T04:01:50.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819740+0000", 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000294772 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819772+0000", 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2255999999999997e-05 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819808+0000", 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6572000000000002e-05 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819963+0000", 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015485600000000001 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.819991+0000", 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8243e-05 2024-09-16T04:01:50.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.821929+0000", 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00193801 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.821991+0000", 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1498e-05 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 51/33 ECSubWrite(tid=94, reqid=client.4646.0:376, at_version=51'37, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.822264+0000", 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.590512839000001, 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021617670000000002, 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.822264+0000", 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.822264+0000", 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.822265+0000", 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.822362+0000", 2024-09-16T04:01:50.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6824000000000007e-05 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.822390+0000", 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8276000000000002e-05 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.822413+0000", 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.822565+0000", 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000152132 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.822594+0000", 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9536999999999999e-05 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.824364+0000", 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001769631 2024-09-16T04:01:50.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.824426+0000", 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1970999999999993e-05 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 56/33 ECSubWrite(tid=106, reqid=osd.4.0:106, at_version=56'49, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:41.730232+0000", 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.6825453879999994, 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053459069999999996, 2024-09-16T04:01:50.498 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730232+0000", 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730232+0000", 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730232+0000", 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5400000000000002e-07 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730256+0000", 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4460999999999998e-05 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730266+0000", 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8279999999999998e-06 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730277+0000", 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1089e-05 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730413+0000", 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013620200000000001 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730446+0000", 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2895999999999998e-05 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.735500+0000", 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0050538249999999996 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.735577+0000", 2024-09-16T04:01:50.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7352000000000001e-05 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 56/33 ECSubWrite(tid=107, reqid=osd.4.0:107, at_version=56'51, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:41.733013+0000", 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.6797643539999996, 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041949350000000003, 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.733013+0000", 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.733013+0000", 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.733013+0000", 2024-09-16T04:01:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.23e-07 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.733042+0000", 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.917e-05 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.733053+0000", 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0848e-05 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.733062+0000", 2024-09-16T04:01:50.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6449999999999994e-06 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.733232+0000", 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170123 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.733259+0000", 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6823000000000001e-05 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.737111+0000", 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003852149 2024-09-16T04:01:50.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.737207+0000", 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6453999999999995e-05 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.507 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:50.508 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:50.413+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:50.509 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:01:50.518 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:50.508+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:50.531 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:50.618 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:50.617+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:50.639 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:50.639 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:50.639 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:50.640 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:50.656 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:01:50.719 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:50.718+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:50.790 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:01:50.820 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:50.816+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:50.853 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T04:01:50.861 INFO:tasks.rados.rados.0.smithi188.stdout:202: done (1 left) 2024-09-16T04:01:50.861 INFO:tasks.rados.rados.0.smithi188.stdout:203: done (0 left) 2024-09-16T04:01:50.861 INFO:tasks.rados.rados.0.smithi188.stdout:204: copy_from oid 36 from oid 33 current snap is 21 2024-09-16T04:01:50.861 INFO:tasks.rados.rados.0.smithi188.stdout:205: copy_from oid 28 from oid 8 current snap is 21 2024-09-16T04:01:50.861 INFO:tasks.rados.rados.0.smithi188.stdout:206: read oid 48 snap -1 2024-09-16T04:01:50.861 INFO:tasks.rados.rados.0.smithi188.stdout:206: expect (ObjNum 83 snap 10 seq_num 83) 2024-09-16T04:01:50.861 INFO:tasks.rados.rados.0.smithi188.stdout:207: read oid 12 snap -1 2024-09-16T04:01:50.862 INFO:tasks.rados.rados.0.smithi188.stdout:207: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:01:50.862 INFO:tasks.rados.rados.0.smithi188.stdout:208: read oid 35 snap -1 2024-09-16T04:01:50.862 INFO:tasks.rados.rados.0.smithi188.stdout:208: expect (ObjNum 66 snap 1 seq_num 66) 2024-09-16T04:01:50.862 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 19 current snap is 21 2024-09-16T04:01:50.862 INFO:tasks.rados.rados.0.smithi188.stdout:209: seq_num 98 ranges {0=491520,491520=589824,1081344=557056,1638400=16384} 2024-09-16T04:01:50.876 INFO:tasks.rados.rados.0.smithi188.stdout:209: writing smithi18831615-19 from 0 to 491520 tid 1 2024-09-16T04:01:50.881 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T04:01:50.887 INFO:tasks.rados.rados.0.smithi188.stdout:209: writing smithi18831615-19 from 491520 to 1081344 tid 2 2024-09-16T04:01:50.898 INFO:tasks.rados.rados.0.smithi188.stdout:209: writing smithi18831615-19 from 1081344 to 1638400 tid 3 2024-09-16T04:01:50.899 INFO:tasks.rados.rados.0.smithi188.stdout:209: writing smithi18831615-19 from 1638400 to 1654784 tid 4 2024-09-16T04:01:50.899 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 5 current snap is 21 2024-09-16T04:01:50.899 INFO:tasks.rados.rados.0.smithi188.stdout:210: seq_num 99 ranges {81920=507904,589824=606208,1196032=557056} 2024-09-16T04:01:50.909 INFO:tasks.rados.rados.0.smithi188.stdout:210: writing smithi18831615-5 from 81920 to 589824 tid 1 2024-09-16T04:01:50.918 INFO:tasks.rados.rados.0.smithi188.stdout:210: writing smithi18831615-5 from 589824 to 1196032 tid 2 2024-09-16T04:01:50.926 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:50.920+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:50.926 INFO:tasks.rados.rados.0.smithi188.stdout:210: writing smithi18831615-5 from 1196032 to 1753088 tid 3 2024-09-16T04:01:50.926 INFO:tasks.rados.rados.0.smithi188.stdout:211: rollback oid 10 current snap is 21 2024-09-16T04:01:50.926 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 10 to 16 2024-09-16T04:01:50.926 INFO:tasks.rados.rados.0.smithi188.stdout:207: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:01:50.929 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 52.466685810999998, 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.56051925899999999, 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:01:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460473+0000", 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.097e-06 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460488+0000", 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5335000000000001e-05 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460512+0000", 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4108999999999999e-05 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460533+0000", 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0721999999999999e-05 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460655+0000", 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122328 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460689+0000", 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014402+0000", 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.55371241699999996 2024-09-16T04:01:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014725+0000", 2024-09-16T04:01:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000323345 2024-09-16T04:01:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.015144+0000", 2024-09-16T04:01:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041881500000000001 2024-09-16T04:01:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020086+0000", 2024-09-16T04:01:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049418869999999998 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020836+0000", 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074984300000000004 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020926+0000", 2024-09-16T04:01:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0070999999999999e-05 2024-09-16T04:01:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020990+0000", 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4256000000000003e-05 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.514778716000002, 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016442578999999999, 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 114 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412378+0000", 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412379+0000", 2024-09-16T04:01:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.06e-07 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412387+0000", 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1799999999999996e-06 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412397+0000", 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0236e-05 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412404+0000", 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2989999999999999e-06 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.422787+0000", 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010382561 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.422907+0000", 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120316 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.423117+0000", 2024-09-16T04:01:50.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020991200000000001 2024-09-16T04:01:50.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427100+0000", 2024-09-16T04:01:50.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039832779999999998 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428687+0000", 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00158661 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428763+0000", 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5679000000000005e-05 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428821+0000", 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7802e-05 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.51471772, 2024-09-16T04:01:50.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01663504, 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 115 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412439+0000", 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412440+0000", 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.083e-06 2024-09-16T04:01:50.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412538+0000", 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7979999999999994e-05 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412552+0000", 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3736e-05 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412560+0000", 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2630000000000003e-06 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424204+0000", 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011644046 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424328+0000", 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124282 2024-09-16T04:01:50.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.424537+0000", 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020823900000000001 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427111+0000", 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025745849999999999 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428934+0000", 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018232680000000001 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429015+0000", 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1034999999999999e-05 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429074+0000", 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8523000000000002e-05 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.952 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.514464341, 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016628581, 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.953 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 116 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412692+0000", 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412693+0000", 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.077e-06 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412701+0000", 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8390000000000007e-06 2024-09-16T04:01:50.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412715+0000", 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3482999999999999e-05 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412726+0000", 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0798999999999999e-05 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425538+0000", 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012812235999999999 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425665+0000", 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000126995 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.425861+0000", 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019642699999999999 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.427642+0000", 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001781244 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429160+0000", 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001517795 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429264+0000", 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010388299999999999 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.429321+0000", 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6802999999999998e-05 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:50.959 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.514416834000002, 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019211259000000001, 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 117 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412741+0000", 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8700000000000005e-07 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412747+0000", 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1929999999999998e-06 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412757+0000", 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0039e-05 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412764+0000", 2024-09-16T04:01:50.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3259999999999998e-06 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426386+0000", 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013621668999999999 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426547+0000", 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000161192 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426753+0000", 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020624599999999999 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428173+0000", 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001419305 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431799+0000", 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036262320000000001 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431890+0000", 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0987000000000001e-05 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431951+0000", 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1383000000000005e-05 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.965 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.161591111, 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016729104000000002, 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.966 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 160 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765567+0000", 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765573+0000", 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0260000000000004e-06 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765583+0000", 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765590+0000", 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4800000000000004e-06 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774469+0000", 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.008879184 2024-09-16T04:01:50.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774666+0000", 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019695699999999999 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774998+0000", 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00033151099999999999 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778132+0000", 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031338849999999999 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782066+0000", 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039342630000000003 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782181+0000", 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000115093 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782295+0000", 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113726 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.161554367000001, 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017211085000000001, 2024-09-16T04:01:50.972 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 161 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:50.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765603+0000", 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765727+0000", 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124184 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765750+0000", 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2296000000000001e-05 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765759+0000", 2024-09-16T04:01:50.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2760000000000006e-06 2024-09-16T04:01:50.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775766+0000", 2024-09-16T04:01:50.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010007085000000001 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775856+0000", 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9683000000000002e-05 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.776003+0000", 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014695999999999999 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779325+0000", 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033221930000000002 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782596+0000", 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032713519999999999 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782726+0000", 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013011500000000001 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782813+0000", 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7174000000000003e-05 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.161379965000002, 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017217072999999999, 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 162 2024-09-16T04:01:50.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765778+0000", 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765784+0000", 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2659999999999998e-06 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765794+0000", 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.029e-05 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765809+0000", 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4474999999999999e-05 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777438+0000", 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011628929 2024-09-16T04:01:50.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777529+0000", 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1577999999999996e-05 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777666+0000", 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013660699999999999 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781937+0000", 2024-09-16T04:01:50.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042713660000000004 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782885+0000", 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00094829699999999996 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782947+0000", 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1261999999999997e-05 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782994+0000", 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7139999999999999e-05 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.161329582, 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01977027, 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.986 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 163 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765828+0000", 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765834+0000", 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7830000000000004e-06 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765844+0000", 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0149000000000001e-05 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765852+0000", 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0199999999999994e-06 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778628+0000", 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012776183999999999 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778715+0000", 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6941000000000007e-05 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778850+0000", 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013517800000000001 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783007+0000", 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041571960000000002 2024-09-16T04:01:50.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785458+0000", 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002450626 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785539+0000", 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1213999999999999e-05 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785597+0000", 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8258999999999999e-05 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.161283904999998, 2024-09-16T04:01:50.993 INFO:tasks.rados.rados.0.smithi188.stdout:209: finishing write tid 1 to smithi18831615-19 2024-09-16T04:01:50.993 INFO:tasks.rados.rados.0.smithi188.stdout:207: done (7 left) 2024-09-16T04:01:50.994 INFO:tasks.rados.rados.0.smithi188.stdout:212: copy_from oid 12 from oid 24 current snap is 21 2024-09-16T04:01:50.994 INFO:tasks.rados.rados.0.smithi188.stdout:209: finishing write tid 2 to smithi18831615-19 2024-09-16T04:01:50.994 INFO:tasks.rados.rados.0.smithi188.stdout:206: expect (ObjNum 83 snap 10 seq_num 83) 2024-09-16T04:01:50.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019887134000000001, 2024-09-16T04:01:50.994 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:50.994 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 164 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765874+0000", 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0130000000000001e-06 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765996+0000", 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122039 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766015+0000", 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9405000000000001e-05 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766025+0000", 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0096e-05 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779427+0000", 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013401569 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779649+0000", 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022229500000000001 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.780098+0000", 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00044856699999999998 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:50.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783018+0000", 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002919792 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785656+0000", 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026383489999999999 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785714+0000", 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8177000000000002e-05 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:50.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785760+0000", 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5831999999999998e-05 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.161112983000002, 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021760347999999999, 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.000 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 165 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:51.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766045+0000", 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766051+0000", 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8140000000000003e-06 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766062+0000", 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0592e-05 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766069+0000", 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3629999999999996e-06 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781249+0000", 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015179349999999999 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781356+0000", 2024-09-16T04:01:51.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000107747 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781496+0000", 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013985500000000001 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783295+0000", 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017986199999999999 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:51.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787679+0000", 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0043837559999999999 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787752+0000", 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3192e-05 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787804+0000", 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2182999999999998e-05 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.006 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.992067386000002, 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015994999999999999, 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 187 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.007 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935089+0000", 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935091+0000", 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.567e-06 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935103+0000", 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1851999999999999e-05 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935127+0000", 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4596e-05 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935142+0000", 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4986e-05 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946804+0000", 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011661985 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.946926+0000", 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000121455 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.947115+0000", 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018956100000000001 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950511+0000", 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033955729999999998 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950887+0000", 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037619500000000001 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951025+0000", 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013776399999999999 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951084+0000", 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9466000000000003e-05 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.991989854000003, 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016322997999999998, 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 188 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935167+0000", 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935168+0000", 2024-09-16T04:01:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4270000000000001e-06 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935319+0000", 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000150383 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935346+0000", 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7750999999999999e-05 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935392+0000", 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5361000000000002e-05 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948392+0000", 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013000429000000001 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948504+0000", 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011150900000000001 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.948716+0000", 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021198300000000001 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951106+0000", 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0023904899999999999 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951365+0000", 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025906300000000002 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951438+0000", 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2571999999999996e-05 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951490+0000", 2024-09-16T04:01:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2030000000000002e-05 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.991730240999999, 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016713338000000001, 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 189 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:01:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935428+0000", 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935443+0000", 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4599e-05 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935468+0000", 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4788000000000001e-05 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935486+0000", 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8249000000000001e-05 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949701+0000", 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014214839 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949818+0000", 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011776499999999999 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950001+0000", 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018260100000000001 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951914+0000", 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001913049 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952007+0000", 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3006999999999999e-05 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952083+0000", 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5722000000000004e-05 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952140+0000", 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7006000000000003e-05 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:346 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.922867+0000", 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.004289603, 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016365271000000001, 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.025 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 346 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922867+0000", 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922867+0000", 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922870+0000", 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9059999999999998e-06 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922886+0000", 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5603e-05 2024-09-16T04:01:51.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922921+0000", 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5583000000000001e-05 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922940+0000", 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8885999999999998e-05 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923091+0000", 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015139099999999999 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923104+0000", 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2924e-05 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.926969+0000", 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003865074 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.927220+0000", 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025061199999999999 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.939232+0000", 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012012292000000001 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:347 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:51.030 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.004175733, 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018664877, 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 347 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922982+0000", 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.457e-06 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922991+0000", 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923032+0000", 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0878999999999999e-05 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923051+0000", 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9406e-05 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923225+0000", 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000173781 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923244+0000", 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8583000000000001e-05 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928577+0000", 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053332600000000003 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928836+0000", 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025873200000000002 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.941646+0000", 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01281 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:348 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:51.035 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.004042554000002, 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020600073999999999, 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 348 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923116+0000", 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8530000000000001e-06 2024-09-16T04:01:51.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923127+0000", 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1396000000000001e-05 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923152+0000", 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4253e-05 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923168+0000", 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6685999999999999e-05 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923332+0000", 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016361899999999999 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923353+0000", 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0894999999999999e-05 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929889+0000", 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0065358930000000001 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.930145+0000", 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000256452 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.943714+0000", 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013569027000000001 2024-09-16T04:01:51.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:477 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1852175550000004, 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017902273, 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:51.041 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 477 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741942+0000", 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8830000000000002e-06 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741957+0000", 2024-09-16T04:01:51.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4922999999999999e-05 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741991+0000", 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4094999999999998e-05 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742009+0000", 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7958999999999999e-05 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742143+0000", 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013377699999999999 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742155+0000", 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2692e-05 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.747863+0000", 2024-09-16T04:01:51.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057079360000000003 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.748115+0000", 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025199499999999998 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.759841+0000", 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011726013 2024-09-16T04:01:51.046 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:478 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1851059839999998, 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020150529, 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:51.047 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 478 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742052+0000", 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.423e-06 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742061+0000", 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8370000000000003e-06 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742084+0000", 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3034999999999999e-05 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742100+0000", 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5798999999999999e-05 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742319+0000", 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000218789 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742340+0000", 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1069000000000001e-05 2024-09-16T04:01:51.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.750020+0000", 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0076799820000000001 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.750277+0000", 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025761000000000002 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.762201+0000", 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011923985 2024-09-16T04:01:51.051 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:479 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1850239680000003, 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021828410999999999, 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:01:51.052 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 479 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742134+0000", 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2929999999999999e-06 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742143+0000", 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7569999999999993e-06 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742163+0000", 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9839000000000001e-05 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742176+0000", 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3782e-05 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742460+0000", 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00028378600000000001 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742471+0000", 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0856000000000001e-05 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.753628+0000", 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011156981999999999 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.753886+0000", 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025786400000000001 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.763961+0000", 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010075252 2024-09-16T04:01:51.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.057 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:51.057 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:51.057 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:51.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:01:51.059 INFO:tasks.rados.rados.0.smithi188.stdout:204: finishing copy_from racing read to smithi18831615-36 2024-09-16T04:01:51.059 INFO:tasks.rados.rados.0.smithi188.stdout:208: expect (ObjNum 66 snap 1 seq_num 66) 2024-09-16T04:01:51.059 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:51.052+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:204: finishing copy_from to smithi18831615-36 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 36 v 36 (ObjNum 91 snap 16 seq_num 91) dirty exists 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:209: finishing write tid 3 to smithi18831615-19 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:205: finishing copy_from racing read to smithi18831615-28 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:204: done (7 left) 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:206: done (6 left) 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:208: done (5 left) 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:213: delete oid 9 current snap is 21 2024-09-16T04:01:51.060 INFO:tasks.rados.rados.0.smithi188.stdout:209: finishing write tid 4 to smithi18831615-19 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:205: finishing copy_from to smithi18831615-28 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 28 v 30 (ObjNum 88 snap 16 seq_num 88) dirty exists 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:209: finishing write tid 5 to smithi18831615-19 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:209: finishing write tid 6 to smithi18831615-19 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:209: oid 19 updating version 0 to 27 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:209: oid 19 version 27 is already newer than 24 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:209: oid 19 version 27 is already newer than 23 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:209: oid 19 version 27 is already newer than 25 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:209: oid 19 version 27 is already newer than 26 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 19 v 27 (ObjNum 98 snap 21 seq_num 98) dirty exists 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:209: left oid 19 (ObjNum 98 snap 21 seq_num 98) 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:210: finishing write tid 1 to smithi18831615-5 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:210: finishing write tid 2 to smithi18831615-5 2024-09-16T04:01:51.061 INFO:tasks.rados.rados.0.smithi188.stdout:211: finishing rollback tid 0 to smithi18831615-10 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:211: finishing rollback tid 1 to smithi18831615-10 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:210: finishing write tid 3 to smithi18831615-5 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:210: finishing write tid 4 to smithi18831615-5 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:210: finishing write tid 5 to smithi18831615-5 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:210: oid 5 updating version 0 to 54 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:210: oid 5 updating version 54 to 56 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:210: oid 5 version 56 is already newer than 53 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:210: oid 5 version 56 is already newer than 55 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 5 v 56 (ObjNum 99 snap 21 seq_num 99) dirty exists 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:210: left oid 5 (ObjNum 99 snap 21 seq_num 99) 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:211: finishing rollback tid 2 to smithi18831615-10 2024-09-16T04:01:51.062 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 10 v 31 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T04:01:51.063 INFO:tasks.rados.rados.0.smithi188.stdout:212: finishing copy_from racing read to smithi18831615-12 2024-09-16T04:01:51.063 INFO:tasks.rados.rados.0.smithi188.stdout:212: finishing copy_from to smithi18831615-12 2024-09-16T04:01:51.063 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 12 v 116 (ObjNum 70 snap 3 seq_num 70) dirty exists 2024-09-16T04:01:51.068 INFO:tasks.rados.rados.0.smithi188.stdout:205: done (5 left) 2024-09-16T04:01:51.069 INFO:tasks.rados.rados.0.smithi188.stdout:209: done (4 left) 2024-09-16T04:01:51.069 INFO:tasks.rados.rados.0.smithi188.stdout:210: done (3 left) 2024-09-16T04:01:51.069 INFO:tasks.rados.rados.0.smithi188.stdout:211: done (2 left) 2024-09-16T04:01:51.069 INFO:tasks.rados.rados.0.smithi188.stdout:212: done (1 left) 2024-09-16T04:01:51.069 INFO:tasks.rados.rados.0.smithi188.stdout:213: done (0 left) 2024-09-16T04:01:51.069 INFO:tasks.rados.rados.0.smithi188.stdout:214: copy_from oid 20 from oid 27 current snap is 21 2024-09-16T04:01:51.069 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 12 current snap is 21 2024-09-16T04:01:51.069 INFO:tasks.rados.rados.0.smithi188.stdout:215: seq_num 101 ranges {1572864=540672,2113536=507904,2621440=622592,3244032=32768} 2024-09-16T04:01:51.107 INFO:tasks.rados.rados.0.smithi188.stdout:215: writing smithi18831615-12 from 1572864 to 2113536 tid 1 2024-09-16T04:01:51.115 INFO:tasks.rados.rados.0.smithi188.stdout:215: writing smithi18831615-12 from 2113536 to 2621440 tid 2 2024-09-16T04:01:51.123 INFO:tasks.rados.rados.0.smithi188.stdout:215: writing smithi18831615-12 from 2621440 to 3244032 tid 3 2024-09-16T04:01:51.124 INFO:tasks.rados.rados.0.smithi188.stdout:215: writing smithi18831615-12 from 3244032 to 3276800 tid 4 2024-09-16T04:01:51.124 INFO:tasks.rados.rados.0.smithi188.stdout:216: snap_remove snap 16 2024-09-16T04:01:51.160 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:51.157+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:51.192 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:51.192 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:51.192 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:51.192 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:51.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:01:51.260 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:51.252+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:51.321 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:51.321 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:51.321 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:51.321 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:51.322 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:51.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:01:51.361 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:51.360+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:51.455 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:51.455 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:51.455 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:51.455 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:01:51.455 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:54 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'32, mlcod=25'29)", 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 52.324384025999997, 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005928254, 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.456 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128768+0000", 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128770+0000", 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128784+0000", 2024-09-16T04:01:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3959e-05 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128819+0000", 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5237999999999997e-05 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128838+0000", 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8754999999999999e-05 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.458 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.128996+0000", 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000158419 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.129033+0000", 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6905e-05 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.134608+0000", 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055745869999999998 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.459 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.134697+0000", 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8701000000000003e-05 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:87 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'57, mlcod=25'51)", 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 52.254900315999997, 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056693940000000003, 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.460 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198252+0000", 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.461 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198253+0000", 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2699999999999999e-06 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198371+0000", 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000117349 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198394+0000", 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3768e-05 2024-09-16T04:01:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198411+0000", 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6186000000000001e-05 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198538+0000", 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000127842 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.463 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.198574+0000", 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5451000000000003e-05 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.203863+0000", 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052887919999999996 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.203921+0000", 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8736000000000001e-05 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.464 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:121 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'83, mlcod=25'78)", 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 52.016260719999998, 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0070855379999999997, 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.465 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436892+0000", 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436894+0000", 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8700000000000001e-06 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.466 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436963+0000", 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9152000000000005e-05 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.436991+0000", 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8388e-05 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.437006+0000", 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4883e-05 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.467 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.438211+0000", 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0012053599999999999 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.438267+0000", 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5810000000000003e-05 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.443897+0000", 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056298700000000004 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.443977+0000", 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0204999999999995e-05 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:129 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'87, mlcod=25'85)", 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:51.469 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.859907608, 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054743200000000004, 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593245+0000", 2024-09-16T04:01:51.470 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593246+0000", 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6759999999999999e-06 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593258+0000", 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1715999999999999e-05 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593286+0000", 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7912e-05 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593299+0000", 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3352e-05 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593426+0000", 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000127002 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.472 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.593482+0000", 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5952999999999998e-05 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598617+0000", 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051345549999999998 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598719+0000", 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000102154 2024-09-16T04:01:51.473 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:131 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'89, mlcod=25'85)", 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.858485021999996, 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058446720000000004, 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.474 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594667+0000", 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594669+0000", 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0700000000000001e-06 2024-09-16T04:01:51.475 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594744+0000", 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4637999999999999e-05 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594769+0000", 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5245e-05 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.476 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594782+0000", 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2840000000000001e-05 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594871+0000", 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.9091000000000005e-05 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594901+0000", 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9751e-05 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.477 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600445+0000", 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055442649999999996 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600512+0000", 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6772000000000004e-05 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.478 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:132 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'90, mlcod=25'85)", 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.858338158000002, 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005744519, 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.479 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594814+0000", 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594815+0000", 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2900000000000002e-07 2024-09-16T04:01:51.480 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594825+0000", 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5640000000000002e-06 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594836+0000", 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1025e-05 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594844+0000", 2024-09-16T04:01:51.481 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8699999999999998e-06 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595492+0000", 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00064781699999999997 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595549+0000", 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6842999999999998e-05 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600518+0000", 2024-09-16T04:01:51.482 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004968386 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600559+0000", 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1185e-05 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:133 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'91, mlcod=25'85)", 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:51.483 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.858284402000002, 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057530580000000001, 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.484 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594868+0000", 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.37e-07 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594874+0000", 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3890000000000004e-06 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.485 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594884+0000", 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8970000000000003e-06 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.594890+0000", 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.5989999999999998e-06 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596024+0000", 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0011332289999999999 2024-09-16T04:01:51.486 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596076+0000", 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2253000000000003e-05 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600563+0000", 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004487301 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.487 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.600621+0000", 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7853000000000003e-05 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:134 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'92, mlcod=25'85)", 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.857802163999999, 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0068484870000000003, 2024-09-16T04:01:51.488 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595350+0000", 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.489 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595352+0000", 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595414+0000", 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1926000000000003e-05 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595439+0000", 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4794999999999999e-05 2024-09-16T04:01:51.490 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.595453+0000", 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4283e-05 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596809+0000", 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001356407 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.491 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596847+0000", 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7731000000000002e-05 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602111+0000", 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052641040000000004 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602199+0000", 2024-09-16T04:01:51.492 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7551000000000003e-05 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:135 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'93, mlcod=25'85)", 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.857132593999999, 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006226517, 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.493 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596020+0000", 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596021+0000", 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.403e-06 2024-09-16T04:01:51.494 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596033+0000", 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1888999999999999e-05 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596051+0000", 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8091999999999999e-05 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596065+0000", 2024-09-16T04:01:51.495 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4063000000000001e-05 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597461+0000", 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001395568 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597495+0000", 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3824e-05 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.496 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602202+0000", 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0047076369999999998 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602246+0000", 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4041000000000001e-05 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.497 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:136 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'94, mlcod=25'85)", 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.856630437, 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057697900000000003, 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.498 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596522+0000", 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596524+0000", 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.796e-06 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596536+0000", 2024-09-16T04:01:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1963e-05 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596560+0000", 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3855000000000001e-05 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.596572+0000", 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2836e-05 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.500 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.597942+0000", 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0013696769999999999 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.598001+0000", 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8856e-05 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602248+0000", 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042472500000000002 2024-09-16T04:01:51.501 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.602292+0000", 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3557000000000003e-05 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:163 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'117, mlcod=25'112)", 2024-09-16T04:01:51.502 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.674246848000003, 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061538799999999996, 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.503 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778906+0000", 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778907+0000", 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.08e-06 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778932+0000", 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5661999999999999e-05 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.504 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778950+0000", 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7852000000000001e-05 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.778961+0000", 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1333e-05 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.779058+0000", 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6280000000000007e-05 2024-09-16T04:01:51.505 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.779087+0000", 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9505e-05 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.784992+0000", 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059048049999999999 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.506 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.785059+0000", 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7362999999999999e-05 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_repop(client.4258.0:165 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'119, mlcod=25'112)", 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 51.671543358999998, 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056794660000000002, 2024-09-16T04:01:51.507 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781609+0000", 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781611+0000", 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.827e-06 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781629+0000", 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7734999999999999e-05 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.509 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781657+0000", 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8065e-05 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781670+0000", 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2993000000000001e-05 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781828+0000", 2024-09-16T04:01:51.510 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000158436 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.781846+0000", 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7651000000000001e-05 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.787255+0000", 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054093639999999998 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.511 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:00:59.787288+0000", 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3395000000000001e-05 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s1 40/33 ECSubWrite(tid=191, reqid=client.4646.0:318, at_version=40'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.027929+0000", 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.425222956999999, 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0060192860000000004, 2024-09-16T04:01:51.512 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027929+0000", 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.513 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027929+0000", 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.027930+0000", 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7000000000000004e-07 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028366+0000", 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00043583899999999999 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.514 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028399+0000", 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2872e-05 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028416+0000", 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6889e-05 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.515 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028558+0000", 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014243599999999999 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.028574+0000", 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5605999999999999e-05 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033846+0000", 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052723049999999997 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.516 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033949+0000", 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000102569 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 40/33 ECSubWrite(tid=51, reqid=client.4646.0:323, at_version=40'28, trim_to=0'0, roll_forward_to=40'26))", 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.033639+0000", 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.419513064, 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0063318030000000004, 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.517 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033639+0000", 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033639+0000", 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.518 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033641+0000", 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.35e-06 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033692+0000", 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.1409000000000002e-05 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033730+0000", 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8229000000000003e-05 2024-09-16T04:01:51.519 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033752+0000", 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1705e-05 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033936+0000", 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000183604 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.033960+0000", 2024-09-16T04:01:51.520 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4253e-05 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039854+0000", 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058937670000000003 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039971+0000", 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00011748600000000001 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.521 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s1 40/33 ECSubWrite(tid=195, reqid=client.4646.0:321, at_version=0'0, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.039640+0000", 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.413511917000001, 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054764610000000002, 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039640+0000", 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039640+0000", 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039642+0000", 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.903e-06 2024-09-16T04:01:51.523 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039670+0000", 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7279000000000001e-05 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039703+0000", 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3108999999999998e-05 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.524 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039726+0000", 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3238999999999999e-05 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.039952+0000", 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00022652999999999999 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.040026+0000", 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3566999999999995e-05 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.525 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.045041+0000", 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0050146260000000003 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.045117+0000", 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6207999999999999e-05 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.526 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs1 60/33 ECSubWrite(tid=177, reqid=client.4646.0:449, at_version=60'27, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.745416+0000", 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.707736208, 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0070786699999999996, 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745416+0000", 2024-09-16T04:01:51.527 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745416+0000", 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745416+0000", 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.53e-07 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.528 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746021+0000", 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00060466100000000004 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746058+0000", 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7054999999999997e-05 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746085+0000", 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7024000000000001e-05 2024-09-16T04:01:51.529 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746355+0000", 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00026936400000000002 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746383+0000", 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8699000000000001e-05 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:51.530 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.752406+0000", 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0060229719999999997 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.752495+0000", 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8541999999999994e-05 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s2 60/33 ECSubWrite(tid=12, reqid=client.4646.0:458, at_version=60'10, trim_to=0'0, roll_forward_to=41'8))", 2024-09-16T04:01:51.531 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.785043+0000", 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.6681090659999995, 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057712170000000004, 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.785043+0000", 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.532 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.785043+0000", 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.785044+0000", 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8000000000000001e-07 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.785689+0000", 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00064493000000000005 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.533 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.785734+0000", 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4978999999999997e-05 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.785757+0000", 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3666e-05 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.785928+0000", 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017093300000000001 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.534 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.785958+0000", 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0309999999999999e-05 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.790717+0000", 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0047589279999999999 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.790815+0000", 2024-09-16T04:01:51.535 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.7090999999999996e-05 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs1 60/33 ECSubWrite(tid=184, reqid=client.4646.0:463, at_version=60'33, trim_to=0'0, roll_forward_to=60'32))", 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.787597+0000", 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.6655550019999996, 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0078106909999999998, 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.536 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.787597+0000", 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.787597+0000", 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.537 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.787599+0000", 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.72e-06 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.788032+0000", 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00043258400000000002 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.788064+0000", 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2762999999999998e-05 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.538 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.788088+0000", 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3745999999999999e-05 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.788312+0000", 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00022408500000000001 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.788344+0000", 2024-09-16T04:01:51.539 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1396000000000002e-05 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.795326+0000", 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0069824489999999999 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.795408+0000", 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1947999999999995e-05 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.540 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs1 60/33 ECSubWrite(tid=187, reqid=client.4646.0:465, at_version=60'35, trim_to=0'0, roll_forward_to=60'32))", 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.793674+0000", 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.6594782749999997, 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056248640000000003, 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.541 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.793674+0000", 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.793674+0000", 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.793676+0000", 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.81e-06 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.793803+0000", 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012674100000000001 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.793834+0000", 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1165999999999999e-05 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.793856+0000", 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1843000000000001e-05 2024-09-16T04:01:51.543 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.794242+0000", 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00038597499999999999 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.794267+0000", 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5643000000000002e-05 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.544 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.799205+0000", 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0049376410000000004 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.799299+0000", 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.4044999999999998e-05 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.545 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 65/33 ECSubWrite(tid=98, reqid=client.4646.0:512, at_version=65'62, trim_to=0'0, roll_forward_to=63'60))", 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:50.902524+0000", 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.55062824399999999, 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0065165190000000001, 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.902524+0000", 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.546 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.902524+0000", 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.902526+0000", 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.584e-06 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:51.547 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.903208+0000", 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00068212600000000002 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.903242+0000", 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3692999999999997e-05 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.903270+0000", 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8331999999999999e-05 2024-09-16T04:01:51.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.903425+0000", 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015516400000000001 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.903450+0000", 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4817e-05 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.908939+0000", 2024-09-16T04:01:51.549 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054889479999999996 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.909041+0000", 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010185500000000001 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:51.550 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:51.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:01:51.552 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:51.545+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:51.653 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:51.648+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:51.678 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:51.678 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:51.678 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:51.678 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:51.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:01:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T04:01:51.726 INFO:tasks.rados.rados.0.smithi188.stdout:216: done (2 left) 2024-09-16T04:01:51.726 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 37 current snap is 21 2024-09-16T04:01:51.726 INFO:tasks.rados.rados.0.smithi188.stdout:214: finishing copy_from racing read to smithi18831615-20 2024-09-16T04:01:51.726 INFO:tasks.rados.rados.0.smithi188.stdout:214: finishing copy_from to smithi18831615-20 2024-09-16T04:01:51.726 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 20 v 13 (ObjNum 90 snap 16 seq_num 90) dirty exists 2024-09-16T04:01:51.726 INFO:tasks.rados.rados.0.smithi188.stdout:215: finishing write tid 1 to smithi18831615-12 2024-09-16T04:01:51.727 INFO:tasks.rados.rados.0.smithi188.stdout:217: seq_num 102 ranges {3031040=557056,3588096=638976,4227072=524288} 2024-09-16T04:01:51.754 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:51.750+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:51.755 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T04:01:51.784 INFO:tasks.rados.rados.0.smithi188.stdout:217: writing smithi18831615-37 from 3031040 to 3588096 tid 1 2024-09-16T04:01:51.792 INFO:tasks.rados.rados.0.smithi188.stdout:217: writing smithi18831615-37 from 3588096 to 4227072 tid 2 2024-09-16T04:01:51.799 INFO:tasks.rados.rados.0.smithi188.stdout:217: writing smithi18831615-37 from 4227072 to 4751360 tid 3 2024-09-16T04:01:51.799 INFO:tasks.rados.rados.0.smithi188.stdout:214: done (2 left) 2024-09-16T04:01:51.799 INFO:tasks.rados.rados.0.smithi188.stdout:218: snap_create 2024-09-16T04:01:51.799 INFO:tasks.rados.rados.0.smithi188.stdout:215: finishing write tid 2 to smithi18831615-12 2024-09-16T04:01:51.799 INFO:tasks.rados.rados.0.smithi188.stdout:215: finishing write tid 3 to smithi18831615-12 2024-09-16T04:01:51.799 INFO:tasks.rados.rados.0.smithi188.stdout:215: finishing write tid 4 to smithi18831615-12 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:215: finishing write tid 5 to smithi18831615-12 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:215: finishing write tid 6 to smithi18831615-12 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:215: oid 12 updating version 0 to 120 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:215: oid 12 version 120 is already newer than 118 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:215: oid 12 updating version 120 to 122 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:215: oid 12 version 122 is already newer than 119 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:215: oid 12 version 122 is already newer than 121 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 12 v 122 (ObjNum 101 snap 21 seq_num 101) dirty exists 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:215: left oid 12 (ObjNum 101 snap 21 seq_num 101) 2024-09-16T04:01:51.800 INFO:tasks.rados.rados.0.smithi188.stdout:217: finishing write tid 1 to smithi18831615-37 2024-09-16T04:01:51.804 INFO:tasks.rados.rados.0.smithi188.stdout:217: finishing write tid 2 to smithi18831615-37 2024-09-16T04:01:51.805 INFO:tasks.rados.rados.0.smithi188.stdout:217: finishing write tid 3 to smithi18831615-37 2024-09-16T04:01:51.808 INFO:tasks.rados.rados.0.smithi188.stdout:217: finishing write tid 4 to smithi18831615-37 2024-09-16T04:01:51.811 INFO:tasks.rados.rados.0.smithi188.stdout:217: finishing write tid 5 to smithi18831615-37 2024-09-16T04:01:51.812 INFO:tasks.rados.rados.0.smithi188.stdout:217: oid 37 updating version 0 to 130 2024-09-16T04:01:51.812 INFO:tasks.rados.rados.0.smithi188.stdout:217: oid 37 updating version 130 to 133 2024-09-16T04:01:51.812 INFO:tasks.rados.rados.0.smithi188.stdout:217: oid 37 version 133 is already newer than 131 2024-09-16T04:01:51.812 INFO:tasks.rados.rados.0.smithi188.stdout:217: oid 37 version 133 is already newer than 132 2024-09-16T04:01:51.812 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 37 v 133 (ObjNum 102 snap 21 seq_num 102) dirty exists 2024-09-16T04:01:51.812 INFO:tasks.rados.rados.0.smithi188.stdout:217: left oid 37 (ObjNum 102 snap 21 seq_num 102) 2024-09-16T04:01:51.826 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:51.826 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:51.826 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:51.826 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:51.826 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:51.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:01:51.881 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:01:51.884 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:51.882+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:51.956 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:51.953+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:52.007 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:52.007 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:52.007 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:52.007 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=55, reqid=client.4646.0:327, at_version=41'32, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.895757+0000", 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.107975825, 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0075359370000000004, 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.008 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895757+0000", 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895757+0000", 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.009 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895757+0000", 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895828+0000", 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.0853999999999996e-05 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895862+0000", 2024-09-16T04:01:52.010 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3896000000000002e-05 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895887+0000", 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4732999999999999e-05 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.011 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.896051+0000", 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016381699999999999 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.896077+0000", 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6075999999999999e-05 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903206+0000", 2024-09-16T04:01:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0071296729999999996 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903292+0000", 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6098000000000001e-05 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.013 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=56, reqid=client.4646.0:328, at_version=41'33, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.898528+0000", 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.105203969000002, 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0091371349999999994, 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.014 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898528+0000", 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898528+0000", 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898530+0000", 2024-09-16T04:01:52.015 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898581+0000", 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0807999999999999e-05 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898624+0000", 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3244999999999997e-05 2024-09-16T04:01:52.016 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898651+0000", 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6716e-05 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898817+0000", 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016536099999999999 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.017 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898841+0000", 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4796000000000001e-05 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907533+0000", 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0086919360000000008 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.018 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907666+0000", 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000132309 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.bs4 41/33 ECSubWrite(tid=69, reqid=osd.4.0:69, at_version=41'20, trim_to=0'0, roll_forward_to=37'18))", 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.105142710999999, 2024-09-16T04:01:52.019 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079802759999999997, 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.020 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898590+0000", 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.4700000000000001e-07 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898652+0000", 2024-09-16T04:01:52.021 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1468000000000002e-05 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898683+0000", 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1025999999999997e-05 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898705+0000", 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2436e-05 2024-09-16T04:01:52.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898898+0000", 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019302300000000001 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898922+0000", 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4202999999999998e-05 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.023 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906462+0000", 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0075398959999999999 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906570+0000", 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000107577 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.024 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/33 ECSubWrite(tid=10, reqid=osd.5.0:10, at_version=41'8, trim_to=0'0, roll_forward_to=37'6))", 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.901860+0000", 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.101872453999999, 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0097700600000000005, 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901860+0000", 2024-09-16T04:01:52.025 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901860+0000", 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901861+0000", 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:01:52.026 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901920+0000", 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.9755e-05 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901951+0000", 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0289000000000001e-05 2024-09-16T04:01:52.027 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901975+0000", 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3929000000000001e-05 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902135+0000", 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016060099999999999 2024-09-16T04:01:52.028 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902159+0000", 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4139999999999999e-05 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911537+0000", 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093778990000000003 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.029 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911630+0000", 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2783999999999998e-05 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=57, reqid=client.4646.0:329, at_version=41'34, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.902328+0000", 2024-09-16T04:01:52.030 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.101404717000001, 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093967600000000005, 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902328+0000", 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.031 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902328+0000", 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902330+0000", 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7439999999999999e-06 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.032 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902376+0000", 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.5858e-05 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902412+0000", 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5281999999999999e-05 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.033 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902436+0000", 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4326000000000001e-05 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902591+0000", 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015515200000000001 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.034 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902621+0000", 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9816e-05 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911638+0000", 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0090172280000000004 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.035 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911724+0000", 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6353999999999993e-05 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=58, reqid=client.4646.0:330, at_version=41'35, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.907383+0000", 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.096349801999999, 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0087796330000000002, 2024-09-16T04:01:52.036 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907383+0000", 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907383+0000", 2024-09-16T04:01:52.037 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907385+0000", 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0439999999999998e-06 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907467+0000", 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2282999999999998e-05 2024-09-16T04:01:52.038 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907508+0000", 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1242999999999998e-05 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.907535+0000", 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7078999999999999e-05 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.039 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908206+0000", 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00067093399999999998 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908229+0000", 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2875999999999999e-05 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.040 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916072+0000", 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0078425939999999996 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916162+0000", 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.0580000000000004e-05 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.041 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 41/33 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=41'19, trim_to=0'0, roll_forward_to=40'15))", 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.910207+0000", 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.093525528000001, 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079503769999999998, 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910207+0000", 2024-09-16T04:01:52.042 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910207+0000", 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910208+0000", 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0440000000000001e-06 2024-09-16T04:01:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910261+0000", 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2645000000000002e-05 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910283+0000", 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2841999999999999e-05 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.044 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.045 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.910301+0000", 2024-09-16T04:01:52.045 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8076000000000001e-05 2024-09-16T04:01:52.045 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.045 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.045 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.045 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911107+0000", 2024-09-16T04:01:52.045 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00080574399999999995 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.911130+0000", 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2685999999999999e-05 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918072+0000", 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0069422989999999999 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.046 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918157+0000", 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.5041000000000001e-05 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/33 ECSubWrite(tid=60, reqid=osd.14.0:60, at_version=41'39, trim_to=0'0, roll_forward_to=40'30))", 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.914069+0000", 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.089662960999998, 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010458115, 2024-09-16T04:01:52.047 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914069+0000", 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.048 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914069+0000", 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914071+0000", 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9700000000000002e-06 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.049 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914182+0000", 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00011102 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914216+0000", 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3915999999999999e-05 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914240+0000", 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3768999999999999e-05 2024-09-16T04:01:52.050 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914400+0000", 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015951100000000001 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914425+0000", 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4922999999999998e-05 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.051 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924429+0000", 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010004341999999999 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924528+0000", 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8664000000000003e-05 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.052 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 41/33 ECSubWrite(tid=101, reqid=osd.11.0:101, at_version=41'54, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.087362099, 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011980414999999999, 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.053 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.054 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.054 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.054 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:01:52.054 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.054 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.054 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.054 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916371+0000", 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6300000000000004e-07 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916424+0000", 2024-09-16T04:01:52.055 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2497999999999998e-05 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916451+0000", 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7673000000000001e-05 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.056 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916473+0000", 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1573000000000002e-05 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916863+0000", 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00039059000000000001 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916888+0000", 2024-09-16T04:01:52.057 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4629000000000001e-05 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928269+0000", 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011381092000000001 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928351+0000", 2024-09-16T04:01:52.058 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1596999999999998e-05 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.087203844000001, 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015945876000000001, 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.059 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.060 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916530+0000", 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.133e-06 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916571+0000", 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1539000000000003e-05 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.061 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916594+0000", 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2915999999999999e-05 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916613+0000", 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8766e-05 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.062 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918993+0000", 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0023803050000000001 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919007+0000", 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3703000000000001e-05 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932390+0000", 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013382897 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.063 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932474+0000", 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.4617000000000007e-05 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.as5 41/33 ECSubWrite(tid=13, reqid=osd.3.0:13, at_version=41'8, trim_to=0'0, roll_forward_to=38'6))", 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:01:52.064 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.086480061, 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010948411999999999, 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917253+0000", 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7299999999999996e-07 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.066 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917309+0000", 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5988e-05 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917340+0000", 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1421999999999998e-05 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917364+0000", 2024-09-16T04:01:52.067 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3893e-05 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917525+0000", 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016099500000000001 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917551+0000", 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5344999999999999e-05 2024-09-16T04:01:52.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928110+0000", 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010559192 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928201+0000", 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.1003999999999998e-05 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.069 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/33 ECSubWrite(tid=197, reqid=osd.13.0:197, at_version=41'18, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.084661970999999, 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013473318999999999, 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.070 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.071 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919071+0000", 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7e-07 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919120+0000", 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.9595000000000002e-05 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919147+0000", 2024-09-16T04:01:52.072 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6415999999999999e-05 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919176+0000", 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9002000000000001e-05 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919332+0000", 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015599200000000001 2024-09-16T04:01:52.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919355+0000", 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3312e-05 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932433+0000", 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013077961000000001 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932544+0000", 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000110571 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/33 ECSubWrite(tid=199, reqid=osd.13.0:199, at_version=41'20, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.082900978000001, 2024-09-16T04:01:52.075 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013627653999999999, 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.076 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920833+0000", 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.809e-06 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920890+0000", 2024-09-16T04:01:52.077 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6799000000000003e-05 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920921+0000", 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1161999999999997e-05 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920944+0000", 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3198999999999999e-05 2024-09-16T04:01:52.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.926110+0000", 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051653740000000004 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.926136+0000", 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6568999999999999e-05 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.079 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.934366+0000", 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0082292779999999996 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.934459+0000", 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.3463999999999998e-05 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.080 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 41/33 ECSubWrite(tid=25, reqid=osd.15.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=41'19))", 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.923453+0000", 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.080279801, 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093221420000000003, 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.081 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923453+0000", 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923453+0000", 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.082 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923455+0000", 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0829999999999998e-06 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923481+0000", 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6463000000000001e-05 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923909+0000", 2024-09-16T04:01:52.083 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00042800899999999997 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923939+0000", 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9916e-05 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924156+0000", 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000216799 2024-09-16T04:01:52.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.924181+0000", 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4966000000000001e-05 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932674+0000", 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0084933649999999992 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.085 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932775+0000", 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000100541 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.bs4 54/33 ECSubWrite(tid=102, reqid=osd.4.0:102, at_version=54'46, trim_to=0'0, roll_forward_to=52'42))", 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:39.802592+0000", 2024-09-16T04:01:52.086 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 12.201139999, 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0076893040000000001, 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.802592+0000", 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.087 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.802592+0000", 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.802593+0000", 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8000000000000005e-07 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.088 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.802639+0000", 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.5735000000000002e-05 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.802661+0000", 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1855999999999999e-05 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.089 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.802672+0000", 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.146e-05 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.803525+0000", 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000852523 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.803548+0000", 2024-09-16T04:01:52.090 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2826000000000001e-05 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.810194+0000", 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0066467230000000002 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.810282+0000", 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7400999999999999e-05 2024-09-16T04:01:52.091 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 54/33 ECSubWrite(tid=87, reqid=osd.14.0:86, at_version=0'0, trim_to=0'0, roll_forward_to=54'55))", 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:39.814246+0000", 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 12.189486028999999, 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0091998979999999998, 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.092 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.814246+0000", 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.814246+0000", 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.093 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.814248+0000", 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0229999999999999e-06 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.814276+0000", 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7866999999999999e-05 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.094 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.814312+0000", 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5274999999999997e-05 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.814334+0000", 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2943e-05 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.095 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.815038+0000", 2024-09-16T04:01:52.096 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00070341700000000002 2024-09-16T04:01:52.096 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.096 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.096 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.096 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.815064+0000", 2024-09-16T04:01:52.096 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5922e-05 2024-09-16T04:01:52.096 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.823327+0000", 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.008263378 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:39.823446+0000", 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000119073 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.097 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 61/33 ECSubWrite(tid=194, reqid=client.4646.0:472, at_version=61'118, trim_to=0'0, roll_forward_to=60'116))", 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:46.743865+0000", 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.2598678259999998, 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0072714590000000001, 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.098 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.743865+0000", 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.743865+0000", 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.099 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.743866+0000", 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0070000000000001e-06 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744955+0000", 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0010898170000000001 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.100 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744996+0000", 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.0535999999999999e-05 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745021+0000", 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5443e-05 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745184+0000", 2024-09-16T04:01:52.101 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000162819 2024-09-16T04:01:52.102 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.102 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.102 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.102 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745204+0000", 2024-09-16T04:01:52.102 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9857e-05 2024-09-16T04:01:52.102 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.102 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.102 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.103 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.751050+0000", 2024-09-16T04:01:52.103 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058462690000000003 2024-09-16T04:01:52.103 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.103 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.103 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.103 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.751136+0000", 2024-09-16T04:01:52.103 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.5711000000000007e-05 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.bs4 66/33 ECSubWrite(tid=140, reqid=osd.4.0:140, at_version=66'67, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.729336+0000", 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.274396525, 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0080354629999999996, 2024-09-16T04:01:52.104 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.729336+0000", 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.105 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.729336+0000", 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.729336+0000", 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8000000000000001e-07 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.106 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.729366+0000", 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9570999999999999e-05 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.729381+0000", 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5474000000000001e-05 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.107 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.729393+0000", 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1950000000000001e-05 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.729464+0000", 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1160000000000003e-05 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.108 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.729520+0000", 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5893000000000002e-05 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.737324+0000", 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0078041710000000004 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.109 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.737371+0000", 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.6863999999999997e-05 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 66/33 ECSubWrite(tid=30, reqid=osd.15.0:30, at_version=66'25, trim_to=0'0, roll_forward_to=65'23))", 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.733696+0000", 2024-09-16T04:01:52.110 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.27003679600000002, 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0092114459999999999, 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.733696+0000", 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.111 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.733696+0000", 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.733696+0000", 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.9099999999999999e-07 2024-09-16T04:01:52.112 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.733717+0000", 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1038000000000001e-05 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.733731+0000", 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3859999999999999e-05 2024-09-16T04:01:52.113 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.733741+0000", 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8570000000000007e-06 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.733800+0000", 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.9648999999999998e-05 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.114 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.733811+0000", 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0845999999999999e-05 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.742869+0000", 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0090580680000000007 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.115 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.742907+0000", 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7737000000000001e-05 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 66/33 ECSubWrite(tid=218, reqid=osd.11.0:218, at_version=66'128, trim_to=0'0, roll_forward_to=63'126))", 2024-09-16T04:01:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.740301+0000", 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.26343109799999997, 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0074836410000000001, 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.740301+0000", 2024-09-16T04:01:52.117 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.740301+0000", 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.740301+0000", 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8400000000000001e-07 2024-09-16T04:01:52.118 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.740326+0000", 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4519999999999999e-05 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.740336+0000", 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0475999999999999e-05 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.119 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.740345+0000", 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1650000000000006e-06 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.740413+0000", 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8763999999999994e-05 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.740427+0000", 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.358e-05 2024-09-16T04:01:52.120 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.747730+0000", 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0073032669999999996 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.747785+0000", 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.4685000000000002e-05 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.121 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.122 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.122 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.122 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.122 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:52.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:01:52.124 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:52.121+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:52.224 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:52.218+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:52.250 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:52.250 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:52.250 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:52.251 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:52.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:01:52.325 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:52.316+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:52.426 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:52.422+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:52.433 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:52.434 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:52.434 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:52.434 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:52.434 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:52.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:01:52.526 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:52.524+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:52.600 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:52.615 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:52.615 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:01:52.615 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 27.723443695, 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062540319999999996, 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:01:52.617 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.13e-07 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889239+0000", 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000639432 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889269+0000", 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0131999999999999e-05 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.618 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889293+0000", 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4396000000000002e-05 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889486+0000", 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019278000000000001 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889535+0000", 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8442000000000001e-05 2024-09-16T04:01:52.619 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894775+0000", 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052406800000000002 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894853+0000", 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7557000000000003e-05 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.620 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 40/33 ECSubWrite(tid=85, reqid=client.4646.0:297, at_version=40'45, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:25.907031+0000", 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 26.705011544000001, 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0050522809999999996, 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907031+0000", 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.621 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907031+0000", 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907090+0000", 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8811000000000003e-05 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907534+0000", 2024-09-16T04:01:52.622 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000443759 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907571+0000", 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7509000000000003e-05 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907594+0000", 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3394999999999999e-05 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907748+0000", 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000153659 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.907778+0000", 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0349e-05 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.912008+0000", 2024-09-16T04:01:52.624 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042299080000000001 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:25.912083+0000", 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4890999999999998e-05 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/33 ECSubWrite(tid=70, reqid=osd.4.0:70, at_version=41'12, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.711722260999998, 2024-09-16T04:01:52.625 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059225500000000004, 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.626 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900322+0000", 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.894e-06 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900375+0000", 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2994000000000001e-05 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.627 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900405+0000", 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9543000000000002e-05 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900447+0000", 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.2824999999999997e-05 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902403+0000", 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001955666 2024-09-16T04:01:52.628 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902429+0000", 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5783e-05 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906151+0000", 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037224810000000001 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906243+0000", 2024-09-16T04:01:52.629 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.1364000000000001e-05 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.ds3 41/33 ECSubWrite(tid=29, reqid=osd.7.0:29, at_version=41'19, trim_to=0'0, roll_forward_to=38'15))", 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.703407862999999, 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053361119999999996, 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.630 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908637+0000", 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9269999999999999e-06 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908688+0000", 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908719+0000", 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1479000000000001e-05 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908741+0000", 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2152999999999999e-05 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908934+0000", 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019235000000000001 2024-09-16T04:01:52.633 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908957+0000", 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3572000000000001e-05 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.913883+0000", 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0049257820000000001 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.634 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.913971+0000", 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7854000000000003e-05 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/33 ECSubWrite(tid=101, reqid=osd.11.0:101, at_version=41'54, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.698025654999999, 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062086830000000004, 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.635 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.636 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7999999999999995e-07 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914081+0000", 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.3151000000000006e-05 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914110+0000", 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9672e-05 2024-09-16T04:01:52.637 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914133+0000", 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2439e-05 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914712+0000", 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00057882500000000004 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914737+0000", 2024-09-16T04:01:52.638 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920147+0000", 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054103859999999997 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920226+0000", 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8573999999999997e-05 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.639 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 41/33 ECSubWrite(tid=100, reqid=osd.11.0:100, at_version=41'21, trim_to=0'0, roll_forward_to=40'19))", 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.697854890999999, 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0078425089999999992, 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:01:52.640 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914189+0000", 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2999999999999999e-07 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.641 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914239+0000", 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0232000000000003e-05 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914261+0000", 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2282000000000001e-05 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914279+0000", 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8283000000000001e-05 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.642 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915647+0000", 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0013673719999999999 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915669+0000", 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2198999999999999e-05 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921948+0000", 2024-09-16T04:01:52.643 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062792710000000003 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922030+0000", 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1940000000000005e-05 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/33 ECSubWrite(tid=72, reqid=osd.4.0:70, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.696993462999998, 2024-09-16T04:01:52.644 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0075338999999999996, 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:01:52.645 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915051+0000", 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.057e-06 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915081+0000", 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9488999999999998e-05 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915114+0000", 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3312000000000002e-05 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915138+0000", 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4450000000000001e-05 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915351+0000", 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00021263900000000001 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.647 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915375+0000", 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4219999999999999e-05 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922495+0000", 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0071197559999999997 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922583+0000", 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7977000000000001e-05 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.648 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.695148883000002, 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0069081630000000002, 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.649 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916895+0000", 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.88e-06 2024-09-16T04:01:52.650 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916949+0000", 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3551999999999998e-05 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916983+0000", 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4178999999999999e-05 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917008+0000", 2024-09-16T04:01:52.651 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5049e-05 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917176+0000", 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016821800000000001 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917201+0000", 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4695999999999999e-05 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.652 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923709+0000", 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0065079020000000003 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923802+0000", 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2687000000000002e-05 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.9s6 51/33 ECSubWrite(tid=75, reqid=client.4646.0:384, at_version=51'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:01:52.653 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.852058+0000", 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.759984396, 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0070891699999999997, 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852058+0000", 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.654 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852058+0000", 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852058+0000", 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3500000000000002e-07 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852200+0000", 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014109499999999999 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852218+0000", 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8380999999999999e-05 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852229+0000", 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1545999999999999e-05 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852297+0000", 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7811000000000005e-05 2024-09-16T04:01:52.656 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852304+0000", 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6250000000000001e-06 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.859119+0000", 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006815354 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.657 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.859147+0000", 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8022999999999999e-05 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.9s6 56/33 ECSubWrite(tid=91, reqid=osd.14.0:91, at_version=56'15, trim_to=0'0, roll_forward_to=54'13))", 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:41.723689+0000", 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.888353802999999, 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0048781789999999998, 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.658 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.723689+0000", 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.723689+0000", 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.723689+0000", 2024-09-16T04:01:52.659 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.5400000000000001e-07 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.723745+0000", 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5658000000000001e-05 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.723790+0000", 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4985999999999999e-05 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.660 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.723815+0000", 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4549000000000002e-05 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.724005+0000", 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019090900000000001 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.724034+0000", 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8580000000000001e-05 2024-09-16T04:01:52.661 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.728471+0000", 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004436929 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:41.728567+0000", 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5914000000000004e-05 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=177, reqid=client.4646.0:449, at_version=60'27, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T04:01:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.745511+0000", 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 6.8665319, 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0067418640000000002, 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745511+0000", 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.663 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745511+0000", 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745511+0000", 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3e-07 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746037+0000", 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00052584300000000001 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.664 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746075+0000", 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7595999999999997e-05 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746101+0000", 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6375000000000001e-05 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746354+0000", 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000253212 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.665 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746382+0000", 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7912999999999999e-05 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.752157+0000", 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057748039999999997 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.752252+0000", 2024-09-16T04:01:52.666 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5390999999999995e-05 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=178, reqid=client.4646.0:450, at_version=60'28, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.749601+0000", 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 6.8624416210000003, 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0049004610000000001, 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.667 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.749601+0000", 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.749601+0000", 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.749603+0000", 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.863e-06 2024-09-16T04:01:52.668 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.750065+0000", 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00046194999999999998 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.750104+0000", 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.9796000000000003e-05 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.669 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.750128+0000", 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3309999999999999e-05 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.750336+0000", 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00020866600000000001 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.750362+0000", 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5471999999999999e-05 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.670 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.754392+0000", 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0040299999999999997 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.754501+0000", 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000109404 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=185, reqid=client.4646.0:464, at_version=60'34, trim_to=0'0, roll_forward_to=60'32))", 2024-09-16T04:01:52.671 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.791419+0000", 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 6.820623737, 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0084625380000000004, 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791419+0000", 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.672 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791419+0000", 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791421+0000", 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.17e-06 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791652+0000", 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023153200000000001 2024-09-16T04:01:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791701+0000", 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8721999999999999e-05 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791726+0000", 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4992000000000001e-05 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.674 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.792082+0000", 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00035561700000000002 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.795537+0000", 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0034547710000000001 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.799810+0000", 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042732020000000003 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.675 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.676 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.799881+0000", 2024-09-16T04:01:52.676 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1532000000000006e-05 2024-09-16T04:01:52.676 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.676 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.676 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.676 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.676 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.676 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=187, reqid=client.4646.0:465, at_version=60'35, trim_to=0'0, roll_forward_to=60'32))", 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.796267+0000", 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 6.8157757710000002, 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0081368080000000006, 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.677 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796267+0000", 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796267+0000", 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796269+0000", 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.678 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796395+0000", 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012676699999999999 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796430+0000", 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4303e-05 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796453+0000", 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3603000000000001e-05 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796862+0000", 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000408568 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796887+0000", 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5109e-05 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.804291+0000", 2024-09-16T04:01:52.680 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0074039270000000003 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.804404+0000", 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000112551 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=189, reqid=client.4646.0:466, at_version=60'36, trim_to=0'0, roll_forward_to=60'32))", 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.796500+0000", 2024-09-16T04:01:52.681 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 6.815542539, 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079727640000000002, 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796500+0000", 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796500+0000", 2024-09-16T04:01:52.682 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796501+0000", 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1000000000000001e-06 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796532+0000", 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0549000000000002e-05 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796556+0000", 2024-09-16T04:01:52.683 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4046e-05 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796584+0000", 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7949999999999998e-05 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.798085+0000", 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001501299 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.684 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.798109+0000", 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4326000000000001e-05 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.804414+0000", 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0063049660000000004 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.804473+0000", 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8527999999999999e-05 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 61/33 ECSubWrite(tid=194, reqid=client.4646.0:472, at_version=61'118, trim_to=0'0, roll_forward_to=60'116))", 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:46.744103+0000", 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.867939711, 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055045090000000003, 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744103+0000", 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.686 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744103+0000", 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744104+0000", 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7300000000000005e-07 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744985+0000", 2024-09-16T04:01:52.687 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00088172800000000003 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745010+0000", 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5179999999999999e-05 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745029+0000", 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8036000000000001e-05 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.688 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745205+0000", 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017667 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745222+0000", 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6776e-05 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.749536+0000", 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0043142340000000001 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.689 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.749607+0000", 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1112000000000006e-05 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/33 ECSubWrite(tid=216, reqid=client.4646.0:540, at_version=65'38, trim_to=0'0, roll_forward_to=60'36))", 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.095329+0000", 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.5167136459999999, 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0068702520000000003, 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.690 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.095329+0000", 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.095329+0000", 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.691 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.095331+0000", 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8989999999999999e-06 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.097846+0000", 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0025153139999999998 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.097874+0000", 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7617999999999999e-05 2024-09-16T04:01:52.692 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.097890+0000", 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6456e-05 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.097998+0000", 2024-09-16T04:01:52.693 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000108162 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.098009+0000", 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.041e-05 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.102158+0000", 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004149224 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.694 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.102199+0000", 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1168999999999998e-05 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 66/33 ECSubWrite(tid=139, reqid=osd.4.0:139, at_version=66'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.727946+0000", 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.88409612699999995, 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0092200370000000004, 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.695 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727946+0000", 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727946+0000", 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.696 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727947+0000", 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8400000000000004e-07 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727980+0000", 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2691000000000003e-05 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727999+0000", 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8987e-05 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.697 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.728012+0000", 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3682e-05 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.728107+0000", 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.4248999999999998e-05 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.728119+0000", 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2194e-05 2024-09-16T04:01:52.698 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.737120+0000", 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0090007349999999993 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.737166+0000", 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.6814999999999998e-05 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.699 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 66/33 ECSubWrite(tid=142, reqid=osd.4.0:139, at_version=0'0, trim_to=0'0, roll_forward_to=66'21))", 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.741702+0000", 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.87034054400000005, 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062051329999999998, 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741702+0000", 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.700 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741702+0000", 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741703+0000", 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0139999999999999e-06 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741717+0000", 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3557000000000001e-05 2024-09-16T04:01:52.701 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741734+0000", 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.753e-05 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741745+0000", 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0451999999999999e-05 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741838+0000", 2024-09-16T04:01:52.702 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.3114000000000003e-05 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741847+0000", 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.064e-06 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.747864+0000", 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006016965 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.703 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.747907+0000", 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3436999999999998e-05 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 66/33 ECSubWrite(tid=222, reqid=client.4646.0:549, at_version=66'130, trim_to=0'0, roll_forward_to=66'128))", 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.788454+0000", 2024-09-16T04:01:52.704 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.82358804100000005, 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052145890000000004, 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.788454+0000", 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.788454+0000", 2024-09-16T04:01:52.705 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.788456+0000", 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4670000000000001e-06 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.788771+0000", 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00031541400000000003 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.706 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.788801+0000", 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9907999999999999e-05 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.788814+0000", 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2951e-05 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.788987+0000", 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017256699999999999 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.707 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.789053+0000", 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.5943000000000002e-05 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.793630+0000", 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0045769419999999996 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.793669+0000", 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.9397000000000002e-05 2024-09-16T04:01:52.708 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.709 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.709 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.709 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:01:52.709 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:01:52.709 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:52.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:01:52.711 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:52.704+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:52.723 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T04:01:52.723 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-09-16T04:01:52.728 INFO:tasks.rados.rados.0.smithi188.stdout:215: done (2 left) 2024-09-16T04:01:52.728 INFO:tasks.rados.rados.0.smithi188.stdout:217: done (1 left) 2024-09-16T04:01:52.729 INFO:tasks.rados.rados.0.smithi188.stdout:218: done (0 left) 2024-09-16T04:01:52.729 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 28 current snap is 22 2024-09-16T04:01:52.729 INFO:tasks.rados.rados.0.smithi188.stdout:219: seq_num 103 ranges {1622016=573440,2195456=786432,2981888=376832} 2024-09-16T04:01:52.750 INFO:tasks.rados.rados.0.smithi188.stdout:219: writing smithi18831615-28 from 1622016 to 2195456 tid 1 2024-09-16T04:01:52.753 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T04:01:52.757 INFO:tasks.rados.rados.0.smithi188.stdout:219: writing smithi18831615-28 from 2195456 to 2981888 tid 2 2024-09-16T04:01:52.761 INFO:tasks.rados.rados.0.smithi188.stdout:219: writing smithi18831615-28 from 2981888 to 3358720 tid 3 2024-09-16T04:01:52.761 INFO:tasks.rados.rados.0.smithi188.stdout:220: rollback oid 41 current snap is 22 2024-09-16T04:01:52.761 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 41 to 8 2024-09-16T04:01:52.761 INFO:tasks.rados.rados.0.smithi188.stdout:221: rollback oid 47 current snap is 22 2024-09-16T04:01:52.762 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 47 to 12 2024-09-16T04:01:52.762 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 29 current snap is 22 2024-09-16T04:01:52.762 INFO:tasks.rados.rados.0.smithi188.stdout:222: seq_num 104 ranges {475136=589824,1064960=540672,1605632=622592} 2024-09-16T04:01:52.769 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:52.769 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:52.772 INFO:tasks.rados.rados.0.smithi188.stdout:222: writing smithi18831615-29 from 475136 to 1064960 tid 1 2024-09-16T04:01:52.778 INFO:tasks.rados.rados.0.smithi188.stdout:222: writing smithi18831615-29 from 1064960 to 1605632 tid 2 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:222: writing smithi18831615-29 from 1605632 to 2228224 tid 3 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:223: snap_create 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:219: finishing write tid 1 to smithi18831615-28 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:219: finishing write tid 2 to smithi18831615-28 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:219: finishing write tid 3 to smithi18831615-28 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:221: finishing rollback tid 0 to smithi18831615-47 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:220: finishing rollback tid 0 to smithi18831615-41 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:221: finishing rollback tid 1 to smithi18831615-47 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:220: finishing rollback tid 1 to smithi18831615-41 2024-09-16T04:01:52.785 INFO:tasks.rados.rados.0.smithi188.stdout:219: finishing write tid 4 to smithi18831615-28 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:220: finishing rollback tid 2 to smithi18831615-41 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 41 v 19 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:221: finishing rollback tid 2 to smithi18831615-47 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 47 v 39 (ObjNum 64 snap 1 seq_num 64) dirty exists 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:222: finishing write tid 1 to smithi18831615-29 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:219: finishing write tid 5 to smithi18831615-28 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:219: oid 28 updating version 0 to 70 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:219: oid 28 version 70 is already newer than 68 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:219: oid 28 updating version 70 to 71 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:219: oid 28 version 71 is already newer than 69 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 28 v 71 (ObjNum 103 snap 22 seq_num 103) dirty exists 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:219: left oid 28 (ObjNum 103 snap 22 seq_num 103) 2024-09-16T04:01:52.786 INFO:tasks.rados.rados.0.smithi188.stdout:222: finishing write tid 2 to smithi18831615-29 2024-09-16T04:01:52.790 INFO:tasks.rados.rados.0.smithi188.stdout:222: finishing write tid 3 to smithi18831615-29 2024-09-16T04:01:52.791 INFO:tasks.rados.rados.0.smithi188.stdout:222: finishing write tid 4 to smithi18831615-29 2024-09-16T04:01:52.793 INFO:tasks.rados.rados.0.smithi188.stdout:222: finishing write tid 5 to smithi18831615-29 2024-09-16T04:01:52.793 INFO:tasks.rados.rados.0.smithi188.stdout:222: oid 29 updating version 0 to 38 2024-09-16T04:01:52.793 INFO:tasks.rados.rados.0.smithi188.stdout:222: oid 29 version 38 is already newer than 36 2024-09-16T04:01:52.793 INFO:tasks.rados.rados.0.smithi188.stdout:222: oid 29 version 38 is already newer than 37 2024-09-16T04:01:52.793 INFO:tasks.rados.rados.0.smithi188.stdout:222: oid 29 version 38 is already newer than 35 2024-09-16T04:01:52.793 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 29 v 38 (ObjNum 104 snap 22 seq_num 104) dirty exists 2024-09-16T04:01:52.793 INFO:tasks.rados.rados.0.smithi188.stdout:222: left oid 29 (ObjNum 104 snap 22 seq_num 104) 2024-09-16T04:01:52.797 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:52.797 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:52.812 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:52.804+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:52.818 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:52.818 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:52.841 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:52.842 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:52.842 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:01:52.842 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:52.854 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:52.855 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:52.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:01:52.875 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:52.896 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:52.896 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:52.912 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:52.904+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:52.916 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:52.917 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:52.950 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:52.950 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:52.966 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:52.966 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:52.978 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:01:52.979 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:01:52.979 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:01:52.979 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:01:52.979 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:01:52.986 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:52.992 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:01:53.006 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:53.006 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:53.014 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:53.008+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:53.027 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:53.027 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:53.043 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:53.043 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:53.064 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:53.064 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:53.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:53.060+0000 7fe4f9ce2640 0 -- 172.21.15.37:0/1995437594 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7fe4f4220f30 msgr2=0x7fe4f4223320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:53.085 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:53.086 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:53.106 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:53.107 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:53.108 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:53.108 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:53.115 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:53.108+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:53.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:01:53.216 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:53.213+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:53.255 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:53.255 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:53.269 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:01:53.317 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:53.314+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:53.418 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:53.413+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:53.434 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:53.434 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:53.448 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:01:53.518 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:53.514+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:53.609 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:53.609 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:53.619 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:53.617+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:53.624 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:01:53.720 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:53.717+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:53.726 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:219: done (4 left) 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:220: done (3 left) 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:221: done (2 left) 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:222: done (1 left) 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:223: done (0 left) 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:224: rollback oid 50 current snap is 23 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 50 to 19 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:225: copy_from oid 7 from oid 49 current snap is 23 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 3 current snap is 23 2024-09-16T04:01:53.733 INFO:tasks.rados.rados.0.smithi188.stdout:226: seq_num 105 ranges {0=606208,606208=688128,1294336=475136} 2024-09-16T04:01:53.738 INFO:tasks.rados.rados.0.smithi188.stdout:226: writing smithi18831615-3 from 0 to 606208 tid 1 2024-09-16T04:01:53.745 INFO:tasks.rados.rados.0.smithi188.stdout:226: writing smithi18831615-3 from 606208 to 1294336 tid 2 2024-09-16T04:01:53.749 INFO:tasks.rados.rados.0.smithi188.stdout:226: writing smithi18831615-3 from 1294336 to 1769472 tid 3 2024-09-16T04:01:53.749 INFO:tasks.rados.rados.0.smithi188.stdout:227: copy_from oid 35 from oid 27 current snap is 23 2024-09-16T04:01:53.749 INFO:tasks.rados.rados.0.smithi188.stdout:224: finishing rollback tid 1 to smithi18831615-50 2024-09-16T04:01:53.749 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 50 v 32 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T04:01:53.749 INFO:tasks.rados.rados.0.smithi188.stdout:224: done (3 left) 2024-09-16T04:01:53.749 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 39 current snap is 23 2024-09-16T04:01:53.749 INFO:tasks.rados.rados.0.smithi188.stdout:228: seq_num 106 ranges {0=622592,622592=442368,1064960=524288,1589248=196608} 2024-09-16T04:01:53.755 INFO:tasks.rados.rados.0.smithi188.stdout:228: writing smithi18831615-39 from 0 to 622592 tid 1 2024-09-16T04:01:53.756 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T04:01:53.761 INFO:tasks.rados.rados.0.smithi188.stdout:228: writing smithi18831615-39 from 622592 to 1064960 tid 2 2024-09-16T04:01:53.767 INFO:tasks.rados.rados.0.smithi188.stdout:228: writing smithi18831615-39 from 1064960 to 1589248 tid 3 2024-09-16T04:01:53.769 INFO:tasks.rados.rados.0.smithi188.stdout:228: writing smithi18831615-39 from 1589248 to 1785856 tid 4 2024-09-16T04:01:53.769 INFO:tasks.rados.rados.0.smithi188.stdout:229: read oid 28 snap -1 2024-09-16T04:01:53.769 INFO:tasks.rados.rados.0.smithi188.stdout:229: expect (ObjNum 103 snap 22 seq_num 103) 2024-09-16T04:01:53.769 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 38 current snap is 23 2024-09-16T04:01:53.769 INFO:tasks.rados.rados.0.smithi188.stdout:230: seq_num 107 ranges {3768320=638976,4407296=458752,4866048=704512} 2024-09-16T04:01:53.805 INFO:tasks.rados.rados.0.smithi188.stdout:230: writing smithi18831615-38 from 3768320 to 4407296 tid 1 2024-09-16T04:01:53.807 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:53.807 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:53.807 INFO:tasks.rados.rados.0.smithi188.stdout:230: writing smithi18831615-38 from 4407296 to 4866048 tid 2 2024-09-16T04:01:53.812 INFO:tasks.rados.rados.0.smithi188.stdout:230: writing smithi18831615-38 from 4866048 to 5570560 tid 3 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:231: copy_from oid 47 from oid 25 current snap is 23 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:226: finishing write tid 1 to smithi18831615-3 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:225: finishing copy_from racing read to smithi18831615-7 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:226: finishing write tid 2 to smithi18831615-3 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:225: finishing copy_from to smithi18831615-7 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 7 v 51 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:226: finishing write tid 3 to smithi18831615-3 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:228: finishing write tid 1 to smithi18831615-39 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:226: finishing write tid 4 to smithi18831615-3 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:228: finishing write tid 2 to smithi18831615-39 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:226: finishing write tid 5 to smithi18831615-3 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:226: oid 3 updating version 0 to 74 2024-09-16T04:01:53.813 INFO:tasks.rados.rados.0.smithi188.stdout:226: oid 3 updating version 74 to 75 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:226: oid 3 version 75 is already newer than 73 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:226: oid 3 version 75 is already newer than 72 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 3 v 75 (ObjNum 105 snap 23 seq_num 105) dirty exists 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:226: left oid 3 (ObjNum 105 snap 23 seq_num 105) 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: finishing write tid 3 to smithi18831615-39 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: finishing write tid 4 to smithi18831615-39 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: finishing write tid 5 to smithi18831615-39 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:227: finishing copy_from racing read to smithi18831615-35 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: finishing write tid 6 to smithi18831615-39 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: oid 39 updating version 0 to 124 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: oid 39 updating version 124 to 126 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: oid 39 version 126 is already newer than 123 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: oid 39 updating version 126 to 127 2024-09-16T04:01:53.814 INFO:tasks.rados.rados.0.smithi188.stdout:228: oid 39 version 127 is already newer than 125 2024-09-16T04:01:53.815 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 39 v 127 (ObjNum 106 snap 23 seq_num 106) dirty exists 2024-09-16T04:01:53.815 INFO:tasks.rados.rados.0.smithi188.stdout:228: left oid 39 (ObjNum 106 snap 23 seq_num 106) 2024-09-16T04:01:53.815 INFO:tasks.rados.rados.0.smithi188.stdout:227: finishing copy_from to smithi18831615-35 2024-09-16T04:01:53.815 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 13 (ObjNum 90 snap 16 seq_num 90) dirty exists 2024-09-16T04:01:53.815 INFO:tasks.rados.rados.0.smithi188.stdout:229: expect (ObjNum 103 snap 22 seq_num 103) 2024-09-16T04:01:53.821 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:53.817+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:53.826 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:01:53.923 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:53.918+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:53.972 INFO:tasks.rados.rados.0.smithi188.stdout:230: finishing write tid 1 to smithi18831615-38 2024-09-16T04:01:53.973 INFO:tasks.rados.rados.0.smithi188.stdout:230: finishing write tid 2 to smithi18831615-38 2024-09-16T04:01:53.973 INFO:tasks.rados.rados.0.smithi188.stdout:230: finishing write tid 3 to smithi18831615-38 2024-09-16T04:01:53.973 INFO:tasks.rados.rados.0.smithi188.stdout:225: done (6 left) 2024-09-16T04:01:53.973 INFO:tasks.rados.rados.0.smithi188.stdout:226: done (5 left) 2024-09-16T04:01:53.973 INFO:tasks.rados.rados.0.smithi188.stdout:227: done (4 left) 2024-09-16T04:01:53.973 INFO:tasks.rados.rados.0.smithi188.stdout:228: done (3 left) 2024-09-16T04:01:53.973 INFO:tasks.rados.rados.0.smithi188.stdout:229: done (2 left) 2024-09-16T04:01:53.973 INFO:tasks.rados.rados.0.smithi188.stdout:232: snap_remove snap 10 2024-09-16T04:01:53.996 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:53.996 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:54.010 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:01:54.023 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:54.018+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:54.124 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:54.121+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:54.132 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:54.132 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:54.146 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:01:54.225 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:54.224+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:54.311 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:54.311 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:54.325 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:01:54.327 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:54.326+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:54.428 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:54.424+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:54.445 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:54.445 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:54.461 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:01:54.530 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:54.526+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:54.585 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:54.585 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:54.599 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:01:54.647 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:54.645+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:54.726 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:54.726 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:54.731 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:01:54.729+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:01:54.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:01:54.751 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:01:54.832 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:54.825+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:54.884 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T04:01:54.896 INFO:tasks.rados.rados.0.smithi188.stdout:232: done (2 left) 2024-09-16T04:01:54.896 INFO:tasks.rados.rados.0.smithi188.stdout:233: read oid 50 snap -1 2024-09-16T04:01:54.896 INFO:tasks.rados.rados.0.smithi188.stdout:233: expect deleted 2024-09-16T04:01:54.896 INFO:tasks.rados.rados.0.smithi188.stdout:234: snap_create 2024-09-16T04:01:54.896 INFO:tasks.rados.rados.0.smithi188.stdout:230: finishing write tid 4 to smithi18831615-38 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:230: finishing write tid 5 to smithi18831615-38 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:230: oid 38 updating version 0 to 138 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:230: oid 38 version 138 is already newer than 136 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:230: oid 38 version 138 is already newer than 135 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:230: oid 38 version 138 is already newer than 137 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 38 v 138 (ObjNum 107 snap 23 seq_num 107) dirty exists 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:230: left oid 38 (ObjNum 107 snap 23 seq_num 107) 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:231: finishing copy_from racing read to smithi18831615-47 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:231: finishing copy_from to smithi18831615-47 2024-09-16T04:01:54.897 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 47 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T04:01:54.916 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:54.916 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:54.917 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T04:01:54.933 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:54.929+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:54.936 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:01:55.034 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:55.029+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:55.095 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:55.095 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:55.109 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:01:55.125 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:01:55.135 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:55.130+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:55.233 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:55.233 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:55.236 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:55.229+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:55.248 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:01:55.293 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:55.294 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:55.315 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:55.315 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:55.335 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:55.336 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:55.337 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:55.334+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:55.357 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:55.357 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:55.365 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:55.366 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:55.366 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:55.366 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:55.366 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:55.375 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:55.380 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:01:55.395 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:55.417 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:55.417 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:55.438 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:55.433+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:55.438 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:55.455 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:55.456 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:55.476 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:55.498 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:55.499 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:55.499 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:55.499 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:55.499 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 50/33 ECSubWrite(tid=86, reqid=client.4646.0:362, at_version=50'30, trim_to=0'0, roll_forward_to=49'28))", 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.748449+0000", 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.747480008, 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.007391313, 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748449+0000", 2024-09-16T04:01:55.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748449+0000", 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748450+0000", 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2999999999999999e-07 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748533+0000", 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3332999999999996e-05 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748568+0000", 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4968999999999997e-05 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748616+0000", 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7376000000000002e-05 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748792+0000", 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017674300000000001 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748819+0000", 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.686e-05 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755744+0000", 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069244069999999996 2024-09-16T04:01:55.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755840+0000", 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6794999999999997e-05 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 50/33 ECSubWrite(tid=87, reqid=client.4646.0:363, at_version=50'31, trim_to=0'0, roll_forward_to=49'28))", 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.752047+0000", 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.743882142, 2024-09-16T04:01:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0055759609999999999, 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752047+0000", 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752047+0000", 2024-09-16T04:01:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752050+0000", 2024-09-16T04:01:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7939999999999998e-06 2024-09-16T04:01:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752097+0000", 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7101999999999997e-05 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752130+0000", 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3683000000000003e-05 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752153+0000", 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2249999999999999e-05 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752390+0000", 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000237235 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752415+0000", 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5147000000000002e-05 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.757530+0000", 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051150259999999999 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.757623+0000", 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2724000000000002e-05 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 51/33 ECSubWrite(tid=147, reqid=client.4646.0:400, at_version=51'79, trim_to=0'0, roll_forward_to=51'77))", 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.886967+0000", 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 18.60896249, 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053040049999999997, 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.886967+0000", 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.886967+0000", 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.886968+0000", 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0729999999999999e-06 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.887015+0000", 2024-09-16T04:01:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7593000000000003e-05 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.887043+0000", 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8215999999999999e-05 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.887063+0000", 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9933e-05 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.887246+0000", 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018294700000000001 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.887272+0000", 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5409999999999999e-05 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.892051+0000", 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0047788140000000002 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.892271+0000", 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022001900000000001 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 54/33 ECSubWrite(tid=165, reqid=osd.11.0:165, at_version=54'100, trim_to=0'0, roll_forward_to=54'94))", 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.798437+0000", 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.697492064, 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0081911970000000008, 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.798437+0000", 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.798437+0000", 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.798438+0000", 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.068e-06 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.798465+0000", 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6818e-05 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.798487+0000", 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1956000000000001e-05 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.798498+0000", 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0757999999999999e-05 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.799456+0000", 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00095809400000000005 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.799466+0000", 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0558e-05 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.806583+0000", 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0071165170000000002 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.806628+0000", 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5427999999999999e-05 2024-09-16T04:01:55.519 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.9s2 54/33 ECSubWrite(tid=85, reqid=osd.14.0:85, at_version=54'13, trim_to=0'0, roll_forward_to=52'11))", 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.799254+0000", 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.696674687, 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010434921, 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.520 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.799254+0000", 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.799254+0000", 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.521 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.799255+0000", 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.799283+0000", 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7186e-05 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.800235+0000", 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00095265300000000003 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.522 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.800248+0000", 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.241e-05 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.800382+0000", 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013391799999999999 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.800392+0000", 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.005e-05 2024-09-16T04:01:55.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809625+0000", 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0092337770000000003 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809689+0000", 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3943999999999996e-05 2024-09-16T04:01:55.524 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 54/33 ECSubWrite(tid=101, reqid=osd.4.0:101, at_version=54'44, trim_to=0'0, roll_forward_to=52'42))", 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.801532+0000", 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.694397503999999, 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0084483240000000001, 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.525 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801532+0000", 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801532+0000", 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.526 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801532+0000", 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.63e-07 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801558+0000", 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6350999999999999e-05 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801571+0000", 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2862e-05 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.527 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801581+0000", 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.927e-06 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801846+0000", 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026477500000000003 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801858+0000", 2024-09-16T04:01:55.528 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1781e-05 2024-09-16T04:01:55.529 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.529 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.529 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809945+0000", 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0080871539999999992 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809980+0000", 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5011000000000001e-05 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 54/33 ECSubWrite(tid=102, reqid=osd.4.0:102, at_version=54'46, trim_to=0'0, roll_forward_to=52'42))", 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.801610+0000", 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.694318857000001, 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011045743, 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801610+0000", 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801610+0000", 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801611+0000", 2024-09-16T04:01:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6600000000000002e-07 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801635+0000", 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3977000000000001e-05 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801646+0000", 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1826e-05 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.533 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801655+0000", 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7860000000000002e-06 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.806826+0000", 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051708489999999999 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.806837+0000", 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1153e-05 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.534 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812609+0000", 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0057714669999999997 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812656+0000", 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7219000000000003e-05 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.535 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds4 54/33 ECSubWrite(tid=35, reqid=client.4646.0:421, at_version=54'25, trim_to=0'0, roll_forward_to=54'23))", 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.812275+0000", 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.683654204, 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0062482470000000002, 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812275+0000", 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.536 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812275+0000", 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812276+0000", 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.085e-06 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812292+0000", 2024-09-16T04:01:55.537 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6030000000000001e-05 2024-09-16T04:01:55.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.538 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.538 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.538 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812335+0000", 2024-09-16T04:01:55.538 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2892000000000002e-05 2024-09-16T04:01:55.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.538 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812345+0000", 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.781e-06 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812443+0000", 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8524999999999996e-05 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812453+0000", 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9450000000000005e-06 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.818476+0000", 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0060227040000000003 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.818523+0000", 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7284999999999999e-05 2024-09-16T04:01:55.540 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 54/33 ECSubWrite(tid=103, reqid=osd.4.0:102, at_version=0'0, trim_to=0'0, roll_forward_to=54'46))", 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.813290+0000", 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.682638737, 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0078145609999999994, 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.541 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.813290+0000", 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.813290+0000", 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.542 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.813291+0000", 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.64e-07 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.813305+0000", 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3966e-05 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.813320+0000", 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4946000000000001e-05 2024-09-16T04:01:55.543 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.813330+0000", 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9429999999999995e-06 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.813390+0000", 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9726999999999999e-05 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.544 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.813400+0000", 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0068000000000001e-05 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.820974+0000", 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0075741690000000004 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.821105+0000", 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013077799999999999 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.545 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 54/33 ECSubWrite(tid=166, reqid=osd.11.0:165, at_version=0'0, trim_to=0'0, roll_forward_to=54'100))", 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.814120+0000", 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.681809128999999, 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069101459999999998, 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.814120+0000", 2024-09-16T04:01:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.814120+0000", 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.814121+0000", 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9299999999999996e-07 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.547 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.814133+0000", 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2115e-05 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.814146+0000", 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.347e-05 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.814155+0000", 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9120000000000004e-06 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.814249+0000", 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3979000000000003e-05 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.814259+0000", 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8709999999999992e-06 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.820979+0000", 2024-09-16T04:01:55.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0067201960000000003 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.821030+0000", 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0710000000000001e-05 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 54/33 ECSubWrite(tid=167, reqid=osd.11.0:167, at_version=54'102, trim_to=0'0, roll_forward_to=54'100))", 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.815726+0000", 2024-09-16T04:01:55.550 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.680202603, 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053332539999999999, 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.815726+0000", 2024-09-16T04:01:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.815726+0000", 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.815727+0000", 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8199999999999999e-07 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.815750+0000", 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2586999999999999e-05 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.815766+0000", 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5999999999999999e-05 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.815775+0000", 2024-09-16T04:01:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2110000000000004e-06 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.815835+0000", 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0359000000000003e-05 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.815845+0000", 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0037999999999999e-05 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.821034+0000", 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051890399999999998 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.821060+0000", 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5236999999999999e-05 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.9s2 56/33 ECSubWrite(tid=93, reqid=osd.14.0:91, at_version=0'0, trim_to=0'0, roll_forward_to=56'15))", 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.733613+0000", 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.762316468, 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069694170000000003, 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733613+0000", 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733613+0000", 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733615+0000", 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1859999999999999e-06 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733648+0000", 2024-09-16T04:01:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3216999999999997e-05 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733679+0000", 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1442000000000002e-05 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733702+0000", 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2157000000000001e-05 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.735916+0000", 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022148580000000001 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.735931+0000", 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4100000000000001e-05 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.740462+0000", 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045314889999999997 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.740582+0000", 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000119968 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 56/33 ECSubWrite(tid=107, reqid=osd.4.0:107, at_version=56'51, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.736750+0000", 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.759178941, 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0060761349999999999, 2024-09-16T04:01:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.736750+0000", 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.736750+0000", 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.736751+0000", 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1069999999999999e-06 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.736805+0000", 2024-09-16T04:01:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3992000000000001e-05 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.736831+0000", 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6197e-05 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.736857+0000", 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5545999999999999e-05 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.737695+0000", 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00083777000000000005 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.737719+0000", 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4496000000000001e-05 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.742745+0000", 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050261810000000002 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.742826+0000", 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0846000000000005e-05 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 56/33 ECSubWrite(tid=169, reqid=osd.11.0:169, at_version=56'104, trim_to=0'0, roll_forward_to=54'102))", 2024-09-16T04:01:55.565 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.796593+0000", 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.699336521999999, 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051397500000000002, 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.796593+0000", 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.566 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.796593+0000", 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.796593+0000", 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5700000000000002e-07 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.796663+0000", 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.9319000000000005e-05 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.796695+0000", 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2870000000000002e-05 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.796719+0000", 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3742000000000001e-05 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.796879+0000", 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000159925 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.796906+0000", 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7299000000000001e-05 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.801630+0000", 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0047233020000000004 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.801732+0000", 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000102636 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 56/33 ECSubWrite(tid=170, reqid=osd.11.0:170, at_version=56'106, trim_to=0'0, roll_forward_to=54'102))", 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.799190+0000", 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.696739402, 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051385490000000001, 2024-09-16T04:01:55.570 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.799190+0000", 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.799190+0000", 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.571 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.572 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.572 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.799191+0000", 2024-09-16T04:01:55.572 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6300000000000001e-06 2024-09-16T04:01:55.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.799229+0000", 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7866e-05 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.799258+0000", 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9238999999999999e-05 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.799274+0000", 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5150000000000001e-05 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.799435+0000", 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016135100000000001 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.799466+0000", 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0725999999999997e-05 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.804232+0000", 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0047666649999999998 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.804328+0000", 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5921999999999994e-05 2024-09-16T04:01:55.575 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/33 ECSubWrite(tid=177, reqid=client.4646.0:449, at_version=60'27, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:45.744788+0000", 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7511408159999995, 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051694059999999997, 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.744788+0000", 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.744788+0000", 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.744789+0000", 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6199999999999997e-07 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745194+0000", 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00040459600000000001 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745218+0000", 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4774e-05 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745236+0000", 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7265000000000002e-05 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745354+0000", 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000118099 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745372+0000", 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7932e-05 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.749903+0000", 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045313519999999998 2024-09-16T04:01:55.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.749958+0000", 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4626000000000002e-05 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 60/33 ECSubWrite(tid=13, reqid=client.4646.0:459, at_version=60'11, trim_to=0'0, roll_forward_to=41'8))", 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:45.790719+0000", 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7052096169999995, 2024-09-16T04:01:55.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0056916299999999996, 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.790719+0000", 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.790719+0000", 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.790721+0000", 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.141e-06 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.791003+0000", 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028228500000000001 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.791020+0000", 2024-09-16T04:01:55.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7408e-05 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.791031+0000", 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0448000000000001e-05 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.791115+0000", 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3956999999999996e-05 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.791127+0000", 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2448e-05 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.796360+0000", 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0052328469999999997 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.796411+0000", 2024-09-16T04:01:55.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1096e-05 2024-09-16T04:01:55.585 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.585 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.585 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.4s4 65/33 ECSubWrite(tid=53, reqid=client.4646.0:536, at_version=65'31, trim_to=0'0, roll_forward_to=63'27))", 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.931937+0000", 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.5639919940000002, 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0054390929999999999, 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931937+0000", 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931937+0000", 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931938+0000", 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2500000000000004e-07 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931958+0000", 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9672000000000001e-05 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931973+0000", 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5809000000000001e-05 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931983+0000", 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6749999999999997e-06 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.932126+0000", 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014294899999999999 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.932154+0000", 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7608999999999999e-05 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.937298+0000", 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051444730000000001 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.937376+0000", 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7980999999999998e-05 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 65/33 ECSubWrite(tid=124, reqid=client.4646.0:532, at_version=0'0, trim_to=0'0, roll_forward_to=65'56))", 2024-09-16T04:01:55.590 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.934553+0000", 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.5613762390000003, 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049451520000000004, 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934553+0000", 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934553+0000", 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934554+0000", 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.186e-06 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934572+0000", 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7864e-05 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934590+0000", 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8291000000000002e-05 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934602+0000", 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1735e-05 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934685+0000", 2024-09-16T04:01:55.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3082000000000002e-05 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934698+0000", 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2636999999999999e-05 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.939448+0000", 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0047508769999999997 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.939498+0000", 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9480000000000001e-05 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 65/33 ECSubWrite(tid=126, reqid=client.4646.0:537, at_version=65'58, trim_to=0'0, roll_forward_to=65'56))", 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.942773+0000", 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.5531559689999996, 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051324659999999996, 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:55.595 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942773+0000", 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942773+0000", 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:55.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942774+0000", 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.378e-06 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942996+0000", 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000221339 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943012+0000", 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5939000000000001e-05 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943020+0000", 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8520000000000004e-06 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943098+0000", 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7966000000000005e-05 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943107+0000", 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6169999999999996e-06 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.947856+0000", 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0047491640000000002 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.947906+0000", 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9211e-05 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:55.599 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:55.601 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:01:55.601 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:55.602 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:55.603 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:55.603 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:55.580+0000 7f4c2cf05640 0 -- 172.21.15.37:0/2789615767 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7f4c28225e80 msgr2=0x7f4c28228270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:55.604 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:55.598+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:55.701 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:55.700+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:55.773 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:55.774 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:55.787 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:01:55.802 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:55.801+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:55.809 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:55.809 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:55.892 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-09-16T04:01:55.892 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:230: done (3 left) 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:231: done (2 left) 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:233: done (1 left) 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:234: done (0 left) 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:235: read oid 11 snap -1 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:235: expect deleted 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:236: rmattr oid 5 current snap is 24 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:236: done (1 left) 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:237: read oid 23 snap -1 2024-09-16T04:01:55.898 INFO:tasks.rados.rados.0.smithi188.stdout:237: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:01:55.899 INFO:tasks.rados.rados.0.smithi188.stdout:238: rollback oid 21 current snap is 24 2024-09-16T04:01:55.899 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 21 to 14 2024-09-16T04:01:55.899 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 44 current snap is 24 2024-09-16T04:01:55.899 INFO:tasks.rados.rados.0.smithi188.stdout:239: seq_num 108 ranges {1540096=655360,2195456=606208,2801664=475136,3276800=81920} 2024-09-16T04:01:55.903 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:55.898+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:55.917 INFO:tasks.rados.rados.0.smithi188.stdout:239: writing smithi18831615-44 from 1540096 to 2195456 tid 1 2024-09-16T04:01:55.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T04:01:55.922 INFO:tasks.rados.rados.0.smithi188.stdout:239: writing smithi18831615-44 from 2195456 to 2801664 tid 2 2024-09-16T04:01:55.926 INFO:tasks.rados.rados.0.smithi188.stdout:239: writing smithi18831615-44 from 2801664 to 3276800 tid 3 2024-09-16T04:01:55.927 INFO:tasks.rados.rados.0.smithi188.stdout:239: writing smithi18831615-44 from 3276800 to 3358720 tid 4 2024-09-16T04:01:55.927 INFO:tasks.rados.rados.0.smithi188.stdout:240: copy_from oid 36 from oid 10 current snap is 24 2024-09-16T04:01:55.928 INFO:tasks.rados.rados.0.smithi188.stdout:241: read oid 42 snap -1 2024-09-16T04:01:55.928 INFO:tasks.rados.rados.0.smithi188.stdout:241: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:55.928 INFO:tasks.rados.rados.0.smithi188.stdout:242: snap_create 2024-09-16T04:01:55.929 INFO:tasks.rados.rados.0.smithi188.stdout:237: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:01:55.944 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:55.944 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:55.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:01:55.959 INFO:tasks.rados.rados.0.smithi188.stdout:238: finishing rollback tid 0 to smithi18831615-21 2024-09-16T04:01:55.959 INFO:tasks.rados.rados.0.smithi188.stdout:238: finishing rollback tid 1 to smithi18831615-21 2024-09-16T04:01:55.959 INFO:tasks.rados.rados.0.smithi188.stdout:238: finishing rollback tid 2 to smithi18831615-21 2024-09-16T04:01:55.959 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 21 v 152 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-09-16T04:01:55.959 INFO:tasks.rados.rados.0.smithi188.stdout:239: finishing write tid 1 to smithi18831615-44 2024-09-16T04:01:55.959 INFO:tasks.rados.rados.0.smithi188.stdout:239: finishing write tid 2 to smithi18831615-44 2024-09-16T04:01:55.959 INFO:tasks.rados.rados.0.smithi188.stdout:240: finishing copy_from racing read to smithi18831615-36 2024-09-16T04:01:55.959 INFO:tasks.rados.rados.0.smithi188.stdout:240: finishing copy_from to smithi18831615-36 2024-09-16T04:01:55.960 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 36 v 31 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T04:01:55.960 INFO:tasks.rados.rados.0.smithi188.stdout:239: finishing write tid 3 to smithi18831615-44 2024-09-16T04:01:55.960 INFO:tasks.rados.rados.0.smithi188.stdout:239: finishing write tid 4 to smithi18831615-44 2024-09-16T04:01:55.960 INFO:tasks.rados.rados.0.smithi188.stdout:239: finishing write tid 5 to smithi18831615-44 2024-09-16T04:01:55.960 INFO:tasks.rados.rados.0.smithi188.stdout:241: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:55.984 INFO:tasks.rados.rados.0.smithi188.stdout:239: finishing write tid 6 to smithi18831615-44 2024-09-16T04:01:55.984 INFO:tasks.rados.rados.0.smithi188.stdout:239: oid 44 updating version 0 to 54 2024-09-16T04:01:55.984 INFO:tasks.rados.rados.0.smithi188.stdout:239: oid 44 version 54 is already newer than 52 2024-09-16T04:01:55.984 INFO:tasks.rados.rados.0.smithi188.stdout:239: oid 44 updating version 54 to 55 2024-09-16T04:01:55.984 INFO:tasks.rados.rados.0.smithi188.stdout:239: oid 44 version 55 is already newer than 51 2024-09-16T04:01:55.984 INFO:tasks.rados.rados.0.smithi188.stdout:239: oid 44 version 55 is already newer than 53 2024-09-16T04:01:55.984 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 44 v 55 (ObjNum 108 snap 24 seq_num 108) dirty exists 2024-09-16T04:01:55.984 INFO:tasks.rados.rados.0.smithi188.stdout:239: left oid 44 (ObjNum 108 snap 24 seq_num 108) 2024-09-16T04:01:56.004 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:55.997+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:56.105 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:56.100+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:352 3.bs0 3:da0cb12f:::smithi18831615-18:head [stat,delete] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:29.922455+0000", 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 26.194335191, 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01139134, 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.121 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 352 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922455+0000", 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922455+0000", 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922457+0000", 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7250000000000002e-06 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922472+0000", 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4278e-05 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922501+0000", 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9179999999999998e-05 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922514+0000", 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2933e-05 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922600+0000", 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6774999999999995e-05 2024-09-16T04:01:56.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.922608+0000", 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9179999999999994e-06 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.925579+0000", 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0029709379999999998 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.925825+0000", 2024-09-16T04:01:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00024584299999999998 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928144+0000", 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0023187759999999998 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933040+0000", 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004896119 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933767+0000", 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00072742599999999998 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933846+0000", 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8429000000000004e-05 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:362 3.bs0 3:d42ef308:::smithi18831615-39:head [append 0~0] snapc b=[b,a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746225+0000", 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.370564797, 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01106538, 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 362 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746225+0000", 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746225+0000", 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746228+0000", 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7530000000000002e-06 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746240+0000", 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2573e-05 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746278+0000", 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7583000000000003e-05 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746306+0000", 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7776e-05 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746478+0000", 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017186999999999999 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746699+0000", 2024-09-16T04:01:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000221683 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748807+0000", 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021077330000000001 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754833+0000", 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0060256429999999998 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.757092+0000", 2024-09-16T04:01:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022589369999999999 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.757290+0000", 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000198829 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:363 3.bs0 3:d42ef308:::smithi18831615-39:head [rollback 5] snapc b=[b,a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.370427840000001, 2024-09-16T04:01:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01339692, 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 363 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746363+0000", 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.55e-06 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746372+0000", 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9900000000000003e-06 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746396+0000", 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.357e-05 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746414+0000", 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8012999999999998e-05 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750473+0000", 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040591439999999998 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750763+0000", 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000290165 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752228+0000", 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014641910000000001 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758077+0000", 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0058492529999999996 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758283+0000", 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020590699999999999 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.759759+0000", 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001476137 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.0s6 51/33 ECSubWrite(tid=72, reqid=client.4646.0:372, at_version=51'50, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.729635+0000", 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.387154820999999, 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0097369429999999996, 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729635+0000", 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729635+0000", 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729636+0000", 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3790000000000001e-06 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729682+0000", 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5626e-05 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729714+0000", 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2376000000000003e-05 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.729737+0000", 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2203000000000001e-05 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730081+0000", 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034408399999999998 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.730106+0000", 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5347e-05 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.739281+0000", 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0091746369999999994 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.739372+0000", 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1291000000000004e-05 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.143 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:374 3.bs0 3:de226c4e:::smithi18831615-16:head [append 0~589824] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.806551+0000", 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.310239157000002, 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0096058089999999999, 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 374 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.806551+0000", 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.806551+0000", 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.806554+0000", 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5329999999999999e-06 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807550+0000", 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00099578300000000008 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807589+0000", 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9446000000000001e-05 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807609+0000", 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9582999999999999e-05 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807760+0000", 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015121399999999999 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.807938+0000", 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017827399999999999 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.809819+0000", 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001880912 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.815226+0000", 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0054064530000000003 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.815343+0000", 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000116888 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.816156+0000", 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081372300000000003 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.148 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:377 3.bs0 3:de226c4e:::smithi18831615-16:head [setxattr _header (57) in=7b] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.819453+0000", 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.297336705999999, 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0091754439999999996, 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.149 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 377 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819453+0000", 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819453+0000", 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819454+0000", 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.082e-06 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819459+0000", 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2220000000000001e-06 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819473+0000", 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4106e-05 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819482+0000", 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9339999999999995e-06 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822129+0000", 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0026468389999999998 2024-09-16T04:01:56.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822264+0000", 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013523300000000001 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822808+0000", 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00054320799999999999 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828019+0000", 2024-09-16T04:01:56.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0052111170000000004 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828531+0000", 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051255599999999995 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828628+0000", 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7146999999999997e-05 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:378 3.bs0 3:de226c4e:::smithi18831615-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.297291853000001, 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.013087019, 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 378 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819499+0000", 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1710000000000001e-06 2024-09-16T04:01:56.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819503+0000", 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9319999999999998e-06 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819519+0000", 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5939000000000001e-05 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819528+0000", 2024-09-16T04:01:56.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9390000000000003e-06 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823625+0000", 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040971779999999999 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823698+0000", 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3417999999999994e-05 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.829121+0000", 2024-09-16T04:01:56.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0054222879999999999 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.829385+0000", 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026470200000000001 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.832585+0000", 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031994520000000002 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.0s6 54/33 ECSubWrite(tid=86, reqid=osd.14.0:86, at_version=54'55, trim_to=0'0, roll_forward_to=52'53))", 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.802606+0000", 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.314183337999999, 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0096979389999999992, 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802606+0000", 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802606+0000", 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802607+0000", 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9799999999999999e-07 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802628+0000", 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1413e-05 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802641+0000", 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2778e-05 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802651+0000", 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9229999999999997e-06 2024-09-16T04:01:56.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802725+0000", 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4374000000000003e-05 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.802737+0000", 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.167e-05 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812215+0000", 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0094780650000000008 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812304+0000", 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9418000000000003e-05 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:423 3.bs0 3:da0cb12f:::smithi18831615-18:head [copy-from ver 9] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:01:56.164 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.312214252, 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.032968130999999998, 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 423 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804577+0000", 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9759999999999998e-06 2024-09-16T04:01:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804585+0000", 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6890000000000004e-06 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804602+0000", 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6912999999999999e-05 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804611+0000", 2024-09-16T04:01:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7600000000000008e-06 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804668+0000", 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7521e-05 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804745+0000", 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6137999999999995e-05 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.168 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.832170+0000", 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.027425393999999999 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834467+0000", 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022970199999999999 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.837427+0000", 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002960074 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.837544+0000", 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011664599999999999 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:424 3.bs0 3:da0cb12f:::smithi18831615-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.312160368000001, 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.029561375000000001, 2024-09-16T04:01:56.170 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 424 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804630+0000", 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6400000000000002e-07 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804633+0000", 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0800000000000002e-06 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804642+0000", 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2520000000000005e-06 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804650+0000", 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3590000000000002e-06 2024-09-16T04:01:56.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804908+0000", 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025827199999999998 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804983+0000", 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5432000000000006e-05 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.833934+0000", 2024-09-16T04:01:56.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.028950975 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834053+0000", 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000118553 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834191+0000", 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013788799999999999 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds6 54/33 ECSubWrite(tid=34, reqid=client.4646.0:420, at_version=54'24, trim_to=0'0, roll_forward_to=42'21))", 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.808861+0000", 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.307929047999998, 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0090378139999999999, 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.808861+0000", 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.808861+0000", 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.808862+0000", 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809209+0000", 2024-09-16T04:01:56.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034700900000000001 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809230+0000", 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1203e-05 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809239+0000", 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3360000000000001e-06 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809356+0000", 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000117198 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.809366+0000", 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6139999999999998e-06 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.817855+0000", 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0084890469999999996 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.817899+0000", 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3531e-05 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:519 3.bs0 3:d947dc37:::smithi18831615-12:head [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.860495+0000", 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.2562946090000002, 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0089320549999999995, 2024-09-16T04:01:56.180 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 519 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.860495+0000", 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.181 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.860495+0000", 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.860496+0000", 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.860501+0000", 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9830000000000004e-06 2024-09-16T04:01:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.860534+0000", 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3315999999999997e-05 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.860547+0000", 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2517000000000001e-05 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.860697+0000", 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014989499999999999 2024-09-16T04:01:56.183 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.860718+0000", 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1321000000000001e-05 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.865655+0000", 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049367960000000002 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.865765+0000", 2024-09-16T04:01:56.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011066899999999999 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.869427+0000", 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036617910000000002 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:532 3.bs0 3:dd44685e:::smithi18831615-5:head [setxattr _header (56) in=7b] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.925712+0000", 2024-09-16T04:01:56.185 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.1910778200000003, 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0086689190000000006, 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 532 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925712+0000", 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925712+0000", 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925713+0000", 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9300000000000006e-07 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925717+0000", 2024-09-16T04:01:56.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8569999999999997e-06 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925725+0000", 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0709999999999995e-06 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925732+0000", 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0219999999999999e-06 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.928569+0000", 2024-09-16T04:01:56.188 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0028374260000000001 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.928699+0000", 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000129324 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.929330+0000", 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00063161000000000001 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.930973+0000", 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016432059999999999 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934300+0000", 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0033263500000000001 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934381+0000", 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.106e-05 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.190 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:533 3.bs0 3:dd44685e:::smithi18831615-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.925744+0000", 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.1910454159999997, 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010438295, 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.191 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 533 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925744+0000", 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925744+0000", 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.192 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925750+0000", 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5879999999999997e-06 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925754+0000", 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6160000000000002e-06 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925764+0000", 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0730999999999999e-05 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925772+0000", 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4120000000000004e-06 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.930745+0000", 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049729609999999997 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.930857+0000", 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000111964 2024-09-16T04:01:56.194 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934783+0000", 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0039266659999999997 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.934878+0000", 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4380000000000001e-05 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.195 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.936183+0000", 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013049769999999999 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:537 3.bs0 3:d947dc37:::smithi18831615-12:head [copy-from ver 116] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.931307+0000", 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.1854831580000003, 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.017397910999999999, 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.196 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 537 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931307+0000", 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.197 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931307+0000", 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931309+0000", 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6400000000000001e-06 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931323+0000", 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3409e-05 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.198 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931352+0000", 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9346999999999999e-05 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931370+0000", 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7819e-05 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931552+0000", 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018190200000000001 2024-09-16T04:01:56.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931712+0000", 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000159918 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942260+0000", 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010547948 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.947604+0000", 2024-09-16T04:01:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053446819999999999 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.948629+0000", 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010242859999999999 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.948705+0000", 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5959999999999998e-05 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:538 3.bs0 3:d947dc37:::smithi18831615-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.931404+0000", 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.1853852969999998, 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012715766, 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 538 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931404+0000", 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931404+0000", 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.203 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931406+0000", 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.31e-06 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931415+0000", 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.166e-06 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931441+0000", 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5643000000000002e-05 2024-09-16T04:01:56.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931457+0000", 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.662e-05 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.932327+0000", 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00087013200000000002 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:01:56.205 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.932390+0000", 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2439000000000003e-05 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943909+0000", 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011519722 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.944011+0000", 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010132999999999999 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.206 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.944120+0000", 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000109404 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:547 3.bs0 3:d947dc37:::smithi18831615-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.124016+0000", 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.9927742090000002, 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010316788, 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.207 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 547 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.124016+0000", 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.208 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.124016+0000", 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.124016+0000", 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.34e-07 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.124018+0000", 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.305e-06 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.209 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.124027+0000", 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9490000000000002e-06 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.124033+0000", 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6509999999999998e-06 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.130067+0000", 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0060340089999999999 2024-09-16T04:01:56.210 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.130199+0000", 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013234 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.132726+0000", 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0025267710000000001 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.211 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.132884+0000", 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000158395 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.134332+0000", 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014479339999999999 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.212 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s2 66/33 ECSubWrite(tid=30, reqid=osd.15.0:30, at_version=66'25, trim_to=0'0, roll_forward_to=65'23))", 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.733839+0000", 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.3829509150000003, 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.009398801, 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.733839+0000", 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.213 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.733839+0000", 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.733840+0000", 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2299999999999999e-07 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.733903+0000", 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3652999999999995e-05 2024-09-16T04:01:56.214 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.733934+0000", 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0749e-05 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.733957+0000", 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2575999999999999e-05 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.734221+0000", 2024-09-16T04:01:56.215 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026395800000000001 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.734247+0000", 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6709000000000001e-05 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.743185+0000", 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0089380099999999997 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.216 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.743238+0000", 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2222999999999998e-05 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.3s2 66/33 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=66'32, trim_to=0'0, roll_forward_to=63'30))", 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.737678+0000", 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.3791120289999999, 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012970337, 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.217 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737678+0000", 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737678+0000", 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737678+0000", 2024-09-16T04:01:56.218 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8299999999999998e-07 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737719+0000", 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0689999999999998e-05 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737735+0000", 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6696000000000001e-05 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.219 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737746+0000", 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0886999999999999e-05 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737879+0000", 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000132767 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737906+0000", 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7336000000000001e-05 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.220 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.750611+0000", 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012704317999999999 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.750648+0000", 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7259999999999999e-05 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.3s2 66/33 ECSubWrite(tid=240, reqid=osd.13.0:240, at_version=66'34, trim_to=0'0, roll_forward_to=63'30))", 2024-09-16T04:01:56.221 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.738724+0000", 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.3780653019999995, 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014283240000000001, 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738724+0000", 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.222 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738724+0000", 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738725+0000", 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4099999999999998e-07 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738747+0000", 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1988999999999999e-05 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.223 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738762+0000", 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4851999999999999e-05 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738770+0000", 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8329999999999994e-06 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738850+0000", 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9889999999999996e-05 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.224 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738866+0000", 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6643999999999999e-05 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.751814+0000", 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01294794 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.753008+0000", 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001193351 2024-09-16T04:01:56.225 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.226 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.226 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.226 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.226 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.226 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:56.227 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:01:56.228 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:56.225+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:56.329 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:01:56.324+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:01:56.392 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:56.392 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:56.392 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:56.392 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:56.406 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:01:56.430 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:01:56.425+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:01:56.532 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:56.525+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:56.573 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:56.574 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:56.588 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:01:56.631 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:56.625+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:56.711 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:296 3.4s0 3:2e4a2a62:::smithi18831615-31:head [rollback 3] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:01:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.886877+0000", 2024-09-16T04:01:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 30.819190653, 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0090995360000000001, 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 296 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886877+0000", 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886877+0000", 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886879+0000", 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886888+0000", 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4889999999999995e-06 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886909+0000", 2024-09-16T04:01:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1540999999999999e-05 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886923+0000", 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3715e-05 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.886997+0000", 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4371999999999999e-05 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:01:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.887004+0000", 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2430000000000002e-06 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.889751+0000", 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027470530000000002 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.889862+0000", 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011124900000000001 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.895977+0000", 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0061149539999999997 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs4 41/33 ECSubWrite(tid=100, reqid=osd.11.0:100, at_version=41'21, trim_to=0'0, roll_forward_to=40'19))", 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.913152+0000", 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 29.792915956999998, 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0065104020000000002, 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913152+0000", 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913152+0000", 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913153+0000", 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913169+0000", 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6121999999999999e-05 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913180+0000", 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1076e-05 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913189+0000", 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2439999999999999e-06 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913234+0000", 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4867999999999998e-05 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.913241+0000", 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3929999999999996e-06 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919631+0000", 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0063907 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919662+0000", 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1167999999999997e-05 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:331 3.4s0 3:2a076d46:::smithi18831615-43:head [copy-from ver 17] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:01:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 29.783616705, 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.031262501999999998, 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 331 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:01:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922453+0000", 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.341e-06 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922458+0000", 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4070000000000001e-06 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922470+0000", 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2363e-05 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922476+0000", 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5380000000000002e-06 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922556+0000", 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0481000000000004e-05 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922641+0000", 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4303999999999998e-05 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.946336+0000", 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.023695544999999998 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953385+0000", 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0070484689999999999 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953631+0000", 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00024578499999999999 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953714+0000", 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3269000000000005e-05 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs4 41/33 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=41'23, trim_to=0'0, roll_forward_to=40'19))", 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.922570+0000", 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 29.783497609000001, 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051287989999999999, 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922570+0000", 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922570+0000", 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922571+0000", 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8400000000000003e-07 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922597+0000", 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5839000000000001e-05 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922610+0000", 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3305999999999999e-05 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922620+0000", 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1649999999999995e-06 2024-09-16T04:01:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922688+0000", 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8880999999999993e-05 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922694+0000", 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9089999999999997e-06 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.927653+0000", 2024-09-16T04:01:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049587800000000003 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.927699+0000", 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6035000000000002e-05 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:332 3.4s0 3:2a076d46:::smithi18831615-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 29.783390474000001, 2024-09-16T04:01:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.026354305000000001, 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 332 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:01:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5900000000000005e-07 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922697+0000", 2024-09-16T04:01:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8867000000000001e-05 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922710+0000", 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2591e-05 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922720+0000", 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0353000000000001e-05 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922817+0000", 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6675000000000005e-05 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922866+0000", 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9060000000000001e-05 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.948791+0000", 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.025925143000000001 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.948905+0000", 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011362300000000001 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949032+0000", 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012703399999999999 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 41/33 ECSubWrite(tid=110, reqid=osd.11.0:109, at_version=0'0, trim_to=0'0, roll_forward_to=41'62))", 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.949009+0000", 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 29.757058609000001, 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.005080574, 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949009+0000", 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949009+0000", 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949010+0000", 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2200000000000003e-07 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949025+0000", 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4595e-05 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949037+0000", 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2404e-05 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949049+0000", 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1673e-05 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949099+0000", 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0135e-05 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949108+0000", 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7139999999999999e-06 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.954059+0000", 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049509330000000002 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.954090+0000", 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1398e-05 2024-09-16T04:01:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:359 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.959867542000001, 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015151255000000001, 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 359 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746203+0000", 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6529999999999998e-06 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746217+0000", 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3946e-05 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746250+0000", 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3269999999999998e-05 2024-09-16T04:01:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746268+0000", 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.804e-05 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746410+0000", 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014167300000000001 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:01:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746432+0000", 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1999999999999999e-05 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750758+0000", 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004325527 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750901+0000", 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000143361 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.761352+0000", 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010450785000000001 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:360 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.959756678000002, 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016051205999999998, 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 360 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:01:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746313+0000", 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3200000000000001e-06 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746326+0000", 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2874000000000001e-05 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746352+0000", 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6395000000000001e-05 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746372+0000", 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9584000000000001e-05 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746569+0000", 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019797599999999999 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746579+0000", 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753659+0000", 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0070800610000000003 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753897+0000", 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023741600000000001 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.762363+0000", 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0084659499999999999 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:361 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.959665673, 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016661775, 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 361 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:01:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746403+0000", 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746412+0000", 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3229999999999998e-06 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746433+0000", 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1484e-05 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746450+0000", 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6252e-05 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746636+0000", 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018645399999999999 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:01:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746645+0000", 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9560000000000003e-06 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754590+0000", 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0079450430000000006 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754769+0000", 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017916000000000001 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.763064+0000", 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0082949830000000006 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.4s0 50/33 ECSubRead(tid=38, to_read={3:20e031d1:::smithi18831615-29:head=0,118784,0}, subchunks={3:20e031d1:::smithi18831615-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.952596731, 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0049784879999999997, 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753471+0000", 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755777+0000", 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0023053320000000002 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.755801+0000", 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4729999999999999e-05 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.756189+0000", 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00038755899999999999 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758450+0000", 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022608670000000002 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs4 60/33 ECSubWrite(tid=177, reqid=client.4646.0:449, at_version=60'27, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:45.744967+0000", 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.961101228, 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053237950000000001, 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.744967+0000", 2024-09-16T04:01:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.744967+0000", 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.744968+0000", 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7300000000000005e-07 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745677+0000", 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00070915099999999999 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745696+0000", 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9695999999999999e-05 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745710+0000", 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3939e-05 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745823+0000", 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011307999999999999 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.745842+0000", 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8615999999999999e-05 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.750240+0000", 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0043980169999999997 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:45.750291+0000", 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0522999999999997e-05 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:534 3.4s0 3:2238ca44:::smithi18831615-10:head [append 0~0] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.925665+0000", 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.7804025990000003, 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0073755740000000002, 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 534 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925665+0000", 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925665+0000", 2024-09-16T04:01:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925668+0000", 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0630000000000002e-06 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925682+0000", 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3766e-05 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925719+0000", 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925743+0000", 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4686e-05 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925862+0000", 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011898199999999999 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925998+0000", 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013553799999999999 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.927248+0000", 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012499329999999999 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.930503+0000", 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032550560000000001 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.932972+0000", 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0024694399999999998 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.933041+0000", 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8616999999999997e-05 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:535 3.4s0 3:2238ca44:::smithi18831615-10:head [rollback 12] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.925797+0000", 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.7802708249999997, 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.007756031, 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 535 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925797+0000", 2024-09-16T04:01:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925797+0000", 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925798+0000", 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2669999999999999e-06 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925854+0000", 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5738000000000001e-05 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925882+0000", 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7480000000000001e-05 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925901+0000", 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8842e-05 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.928067+0000", 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021668490000000002 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.928190+0000", 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000122187 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.928926+0000", 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00073679500000000005 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931721+0000", 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002795119 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.933495+0000", 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0017731419999999999 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.933553+0000", 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8612000000000001e-05 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:536 3.4s0 3:2238ca44:::smithi18831615-10:head [append 0~0] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.925928+0000", 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.7801399709999997, 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011857786, 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 536 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925928+0000", 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925928+0000", 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925930+0000", 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.55e-06 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925937+0000", 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4359999999999996e-06 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925957+0000", 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0435999999999999e-05 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925976+0000", 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.806e-05 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.930543+0000", 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045675589999999997 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.930665+0000", 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000121765 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931251+0000", 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00058618500000000003 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.933728+0000", 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0024773989999999999 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.937681+0000", 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0039529489999999999 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.937786+0000", 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000104447 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 66/33 ECSubWrite(tid=222, reqid=client.4646.0:549, at_version=66'130, trim_to=0'0, roll_forward_to=66'128))", 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.787994+0000", 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.918074056, 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.005242844, 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.787994+0000", 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.787994+0000", 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.787995+0000", 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5499999999999996e-07 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.788587+0000", 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059243900000000005 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.788603+0000", 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5948999999999999e-05 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.788618+0000", 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4819e-05 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.788782+0000", 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000164074 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.788810+0000", 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8022999999999999e-05 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.793199+0000", 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0043892150000000001 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.793237+0000", 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7370000000000003e-05 2024-09-16T04:01:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:566 3.4s0 3:20e031d1:::smithi18831615-29:head [append 0~589824] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:52.772426+0000", 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.9336423119999999, 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0081551909999999991, 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 566 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.772426+0000", 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.772426+0000", 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.772427+0000", 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7e-06 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.773567+0000", 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0011396080000000001 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.773587+0000", 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0477000000000001e-05 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.773599+0000", 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1946999999999999e-05 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.773658+0000", 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8371000000000001e-05 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.773730+0000", 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2380999999999997e-05 2024-09-16T04:01:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.774557+0000", 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082658999999999999 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.776671+0000", 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021138789999999999 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.780520+0000", 2024-09-16T04:01:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0038492919999999998 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.780581+0000", 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0946000000000002e-05 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:567 3.4s0 3:20e031d1:::smithi18831615-29:head [append 0~540672] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:52.777775+0000", 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.9282929289999999, 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050829569999999999, 2024-09-16T04:01:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 567 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.777775+0000", 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.777775+0000", 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.777777+0000", 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9250000000000002e-06 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.779015+0000", 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001238177 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.779032+0000", 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6609999999999999e-05 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.779041+0000", 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.918e-06 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.779102+0000", 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1766000000000005e-05 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.779216+0000", 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000113278 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.779853+0000", 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00063678699999999999 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.780897+0000", 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010442959999999999 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.782788+0000", 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001890994 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.782858+0000", 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0205999999999998e-05 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:568 3.4s0 3:20e031d1:::smithi18831615-29:head [append 0~622592] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:52.783926+0000", 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.9221415660000001, 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0057877429999999997, 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 568 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.783926+0000", 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.783926+0000", 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.783928+0000", 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3400000000000001e-06 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785275+0000", 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001346865 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785287+0000", 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2176e-05 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785295+0000", 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3170000000000002e-06 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785340+0000", 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4774999999999997e-05 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785415+0000", 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4960000000000001e-05 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.786020+0000", 2024-09-16T04:01:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00060502500000000003 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.788111+0000", 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020909679999999999 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.789621+0000", 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015104949999999999 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.789714+0000", 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2822e-05 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:569 3.4s0 3:20e031d1:::smithi18831615-29:head [setxattr _header (57) in=7b] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:52.785311+0000", 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.920757504, 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051349489999999998, 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 569 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785311+0000", 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785311+0000", 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785311+0000", 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0999999999999997e-07 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785321+0000", 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.798e-06 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785330+0000", 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3529999999999995e-06 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785335+0000", 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.789e-06 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.787175+0000", 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0018393420000000001 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.787285+0000", 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000110451 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.787721+0000", 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00043599900000000001 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.788327+0000", 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000605829 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.790349+0000", 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020222170000000002 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.790445+0000", 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6261000000000006e-05 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:570 3.4s0 3:20e031d1:::smithi18831615-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:52.785347+0000", 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 3.920721119, 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0065600240000000002, 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 570 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785347+0000", 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785347+0000", 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785347+0000", 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5300000000000004e-07 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785350+0000", 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8100000000000002e-06 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785358+0000", 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6229999999999997e-06 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.785364+0000", 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6210000000000001e-06 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.787955+0000", 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0025911409999999999 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.788023+0000", 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8087e-05 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.790798+0000", 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027753470000000001 2024-09-16T04:01:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.790882+0000", 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4022000000000003e-05 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:52.791907+0000", 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010248200000000001 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:56.811 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:56.813 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:01:56.813 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:56.809+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:56.894 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T04:01:56.902 INFO:tasks.rados.rados.0.smithi188.stdout:235: done (6 left) 2024-09-16T04:01:56.902 INFO:tasks.rados.rados.0.smithi188.stdout:237: done (5 left) 2024-09-16T04:01:56.903 INFO:tasks.rados.rados.0.smithi188.stdout:238: done (4 left) 2024-09-16T04:01:56.903 INFO:tasks.rados.rados.0.smithi188.stdout:239: done (3 left) 2024-09-16T04:01:56.903 INFO:tasks.rados.rados.0.smithi188.stdout:240: done (2 left) 2024-09-16T04:01:56.903 INFO:tasks.rados.rados.0.smithi188.stdout:241: done (1 left) 2024-09-16T04:01:56.903 INFO:tasks.rados.rados.0.smithi188.stdout:242: done (0 left) 2024-09-16T04:01:56.903 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 8 current snap is 25 2024-09-16T04:01:56.903 INFO:tasks.rados.rados.0.smithi188.stdout:243: seq_num 109 ranges {1622016=671744,2293760=753664,3047424=409600} 2024-09-16T04:01:56.913 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:56.910+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:56.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T04:01:56.925 INFO:tasks.rados.rados.0.smithi188.stdout:243: writing smithi18831615-8 from 1622016 to 2293760 tid 1 2024-09-16T04:01:56.933 INFO:tasks.rados.rados.0.smithi188.stdout:243: writing smithi18831615-8 from 2293760 to 3047424 tid 2 2024-09-16T04:01:56.937 INFO:tasks.rados.rados.0.smithi188.stdout:243: writing smithi18831615-8 from 3047424 to 3457024 tid 3 2024-09-16T04:01:56.937 INFO:tasks.rados.rados.0.smithi188.stdout:244: read oid 16 snap 17 2024-09-16T04:01:56.937 INFO:tasks.rados.rados.0.smithi188.stdout:244: expect (ObjNum 80 snap 10 seq_num 80) 2024-09-16T04:01:56.937 INFO:tasks.rados.rados.0.smithi188.stdout:245: setattr oid 47 current snap is 25 2024-09-16T04:01:56.939 INFO:tasks.rados.rados.0.smithi188.stdout:246: snap_create 2024-09-16T04:01:56.944 INFO:tasks.rados.rados.0.smithi188.stdout:243: finishing write tid 1 to smithi18831615-8 2024-09-16T04:01:56.945 INFO:tasks.rados.rados.0.smithi188.stdout:243: finishing write tid 2 to smithi18831615-8 2024-09-16T04:01:56.946 INFO:tasks.rados.rados.0.smithi188.stdout:243: finishing write tid 3 to smithi18831615-8 2024-09-16T04:01:56.947 INFO:tasks.rados.rados.0.smithi188.stdout:243: finishing write tid 4 to smithi18831615-8 2024-09-16T04:01:56.953 INFO:tasks.rados.rados.0.smithi188.stdout:243: finishing write tid 5 to smithi18831615-8 2024-09-16T04:01:56.953 INFO:tasks.rados.rados.0.smithi188.stdout:243: oid 8 updating version 0 to 52 2024-09-16T04:01:56.953 INFO:tasks.rados.rados.0.smithi188.stdout:243: oid 8 version 52 is already newer than 51 2024-09-16T04:01:56.953 INFO:tasks.rados.rados.0.smithi188.stdout:243: oid 8 version 52 is already newer than 50 2024-09-16T04:01:56.953 INFO:tasks.rados.rados.0.smithi188.stdout:243: oid 8 version 52 is already newer than 49 2024-09-16T04:01:56.953 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 8 v 52 (ObjNum 109 snap 25 seq_num 109) dirty exists 2024-09-16T04:01:56.953 INFO:tasks.rados.rados.0.smithi188.stdout:243: left oid 8 (ObjNum 109 snap 25 seq_num 109) 2024-09-16T04:01:56.954 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:56.954 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:56.954 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:01:56.954 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:56.956 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 47 v 40 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T04:01:56.961 INFO:tasks.rados.rados.0.smithi188.stdout:244: expect (ObjNum 80 snap 10 seq_num 80) 2024-09-16T04:01:56.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:01:57.014 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:57.012+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:57.115 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:01:57.112+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:01:57.132 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:01:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:01:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:01:57.133 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:57.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:01:57.219 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:01:57.218+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.419932178000003, 2024-09-16T04:01:57.313 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0044144730000000004, 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888567+0000", 2024-09-16T04:01:57.314 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.888568+0000", 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.67e-07 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890276+0000", 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0017087719999999999 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.315 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890297+0000", 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0384e-05 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890309+0000", 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1827999999999999e-05 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890402+0000", 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3830000000000001e-05 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.890412+0000", 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8050000000000001e-06 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892961+0000", 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0025483979999999999 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:24.892982+0000", 2024-09-16T04:01:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1189e-05 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 40/33 ECSubWrite(tid=85, reqid=client.4646.0:297, at_version=40'45, trim_to=0'0, roll_forward_to=39'43))", 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:25.906746+0000", 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.401753604, 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004320089, 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906746+0000", 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906746+0000", 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906746+0000", 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9200000000000001e-07 2024-09-16T04:01:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.906973+0000", 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022612000000000001 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.907008+0000", 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5404000000000002e-05 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.907038+0000", 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.044e-05 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.907149+0000", 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000110197 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.907164+0000", 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5673000000000001e-05 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.911000+0000", 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003836023 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:25.911066+0000", 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5740000000000004e-05 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.915429+0000", 2024-09-16T04:01:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 30.39307071, 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0039867210000000004, 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915429+0000", 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915429+0000", 2024-09-16T04:01:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915430+0000", 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5600000000000004e-07 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915455+0000", 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5551e-05 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915472+0000", 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7147e-05 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915482+0000", 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7440000000000002e-06 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915540+0000", 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7469000000000002e-05 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.915545+0000", 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8730000000000004e-06 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919371+0000", 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0038254890000000001 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.919416+0000", 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4592000000000003e-05 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 44/33 ECSubWrite(tid=82, reqid=client.4646.0:352, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:29.928253+0000", 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.380246697, 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050133139999999996, 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928253+0000", 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928253+0000", 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928254+0000", 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928308+0000", 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4018999999999999e-05 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928433+0000", 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012573099999999999 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928459+0000", 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5752999999999999e-05 2024-09-16T04:01:57.329 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928608+0000", 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014835099999999999 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.928619+0000", 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1736000000000001e-05 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933215+0000", 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045959820000000002 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.330 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.933266+0000", 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0881999999999998e-05 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 44/33 ECSubWrite(tid=83, reqid=client.4646.0:352, at_version=0'0, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:29.934164+0000", 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.374335090999999, 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040574779999999998, 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.331 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934164+0000", 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934164+0000", 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.332 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934166+0000", 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8759999999999999e-06 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934195+0000", 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8197000000000001e-05 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934243+0000", 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8899000000000001e-05 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.333 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934267+0000", 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3853e-05 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934410+0000", 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000142387 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.934445+0000", 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5272999999999999e-05 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.334 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.938182+0000", 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0037373789999999999 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:29.938222+0000", 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9613999999999997e-05 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.335 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 49/33 ECSubWrite(tid=84, reqid=osd.4.0:84, at_version=49'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.033687+0000", 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.274812957999998, 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048717830000000002, 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033687+0000", 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.336 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033687+0000", 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033687+0000", 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033747+0000", 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.337 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033793+0000", 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6202999999999998e-05 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033820+0000", 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6461999999999998e-05 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.033999+0000", 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017919400000000001 2024-09-16T04:01:57.338 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.034024+0000", 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5279999999999999e-05 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.038490+0000", 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00446592 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.038558+0000", 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8306000000000006e-05 2024-09-16T04:01:57.339 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 50/33 ECSubWrite(tid=86, reqid=client.4646.0:362, at_version=50'30, trim_to=0'0, roll_forward_to=49'28))", 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.748953+0000", 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 21.559546995000002, 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040299580000000002, 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.340 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748953+0000", 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748953+0000", 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.748953+0000", 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2600000000000002e-07 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.341 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749016+0000", 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2515999999999996e-05 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749052+0000", 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6189000000000002e-05 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749078+0000", 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6106e-05 2024-09-16T04:01:57.342 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749280+0000", 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020179 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.749305+0000", 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5242999999999999e-05 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752877+0000", 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035716670000000002 2024-09-16T04:01:57.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752983+0000", 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000105721 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=90, reqid=osd.4.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=51'33))", 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.733755+0000", 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.574744119999998, 2024-09-16T04:01:57.344 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045050910000000001, 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733755+0000", 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733755+0000", 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733757+0000", 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9630000000000001e-06 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733795+0000", 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7523e-05 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733828+0000", 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3495999999999999e-05 2024-09-16T04:01:57.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.733852+0000", 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3946000000000001e-05 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.734111+0000", 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025884900000000001 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.734137+0000", 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6137000000000001e-05 2024-09-16T04:01:57.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.738170+0000", 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040330920000000003 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.738261+0000", 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0085000000000002e-05 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=93, reqid=client.4646.0:375, at_version=51'36, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.818799+0000", 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.489700800000001, 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0046750480000000002, 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818799+0000", 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818799+0000", 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818800+0000", 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.133e-06 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818967+0000", 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016715400000000001 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.350 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818985+0000", 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7580000000000001e-05 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.818994+0000", 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819109+0000", 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000115531 2024-09-16T04:01:57.351 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819123+0000", 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4173e-05 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823440+0000", 2024-09-16T04:01:57.352 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004317052 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823474+0000", 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3364000000000001e-05 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=94, reqid=client.4646.0:376, at_version=51'37, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.821592+0000", 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.486907585000001, 2024-09-16T04:01:57.353 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0038958069999999998, 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821592+0000", 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821592+0000", 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.354 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821593+0000", 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5999999999999991e-07 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821673+0000", 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0075999999999997e-05 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821700+0000", 2024-09-16T04:01:57.355 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7154000000000001e-05 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821715+0000", 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5167999999999999e-05 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821800+0000", 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4556999999999997e-05 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.356 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.821814+0000", 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3793e-05 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.825444+0000", 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036301319999999999 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.825488+0000", 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3967000000000001e-05 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.357 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.358 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.358 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.358 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.358 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 51/33 ECSubWrite(tid=95, reqid=client.4646.0:377, at_version=51'38, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T04:01:57.358 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.822903+0000", 2024-09-16T04:01:57.358 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.485596406999999, 2024-09-16T04:01:57.359 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053964039999999996, 2024-09-16T04:01:57.359 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822903+0000", 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822903+0000", 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.360 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822905+0000", 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7659999999999999e-06 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.822972+0000", 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6878999999999994e-05 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.361 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823001+0000", 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9346e-05 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823022+0000", 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.12e-05 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823123+0000", 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000100784 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.362 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823136+0000", 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2624e-05 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828258+0000", 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051225280000000003 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.828300+0000", 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1276999999999998e-05 2024-09-16T04:01:57.363 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 56/33 ECSubWrite(tid=106, reqid=osd.4.0:106, at_version=56'49, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.730457+0000", 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.578042644, 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0085327589999999991, 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.364 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730457+0000", 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730457+0000", 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730458+0000", 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4600000000000004e-07 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.365 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730516+0000", 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8106000000000002e-05 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730546+0000", 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0043e-05 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730569+0000", 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2878999999999999e-05 2024-09-16T04:01:57.366 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730837+0000", 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026877999999999998 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730865+0000", 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7236000000000001e-05 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.738863+0000", 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.007998794 2024-09-16T04:01:57.367 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.738990+0000", 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012627499999999999 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 56/33 ECSubWrite(tid=107, reqid=osd.4.0:107, at_version=56'51, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.732929+0000", 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.575570957, 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0092828939999999999, 2024-09-16T04:01:57.368 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.732929+0000", 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.732929+0000", 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.369 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.732930+0000", 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5969999999999999e-06 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.732985+0000", 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4951999999999998e-05 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733014+0000", 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8544e-05 2024-09-16T04:01:57.370 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733036+0000", 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1942000000000001e-05 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.738085+0000", 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050491980000000004 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.371 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.738121+0000", 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6542999999999999e-05 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.742115+0000", 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0039939470000000003 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.742212+0000", 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6170999999999998e-05 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.372 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 56/33 ECSubWrite(tid=172, reqid=osd.11.0:172, at_version=56'108, trim_to=0'0, roll_forward_to=56'106))", 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.807794+0000", 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.500705627, 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0044611299999999998, 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.373 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.807794+0000", 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.807794+0000", 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.807796+0000", 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.824e-06 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.807877+0000", 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1521999999999996e-05 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.807915+0000", 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7333000000000003e-05 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.807937+0000", 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1959000000000001e-05 2024-09-16T04:01:57.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.808073+0000", 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013641799999999999 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.808102+0000", 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8929999999999999e-05 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.376 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.812155+0000", 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040534430000000003 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.812255+0000", 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9701e-05 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 61/33 ECSubWrite(tid=194, reqid=client.4646.0:472, at_version=61'118, trim_to=0'0, roll_forward_to=60'116))", 2024-09-16T04:01:57.377 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:46.743690+0000", 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.564809798000001, 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0042310719999999998, 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.743690+0000", 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.378 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.743690+0000", 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.743691+0000", 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3999999999999999e-06 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.744465+0000", 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00077394599999999997 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.379 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.744489+0000", 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4236999999999999e-05 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.744504+0000", 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4346e-05 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.744634+0000", 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013076099999999999 2024-09-16T04:01:57.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.744645+0000", 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0706999999999999e-05 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.747873+0000", 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003227976 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.381 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:46.747921+0000", 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7698999999999999e-05 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 65/33 ECSubWrite(tid=126, reqid=client.4646.0:537, at_version=65'58, trim_to=0'0, roll_forward_to=65'56))", 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.942640+0000", 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.3658598340000001, 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0052597520000000004, 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.382 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942640+0000", 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942640+0000", 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942642+0000", 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8479999999999999e-06 2024-09-16T04:01:57.383 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942997+0000", 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000355199 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943024+0000", 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6883999999999999e-05 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.384 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943037+0000", 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.327e-05 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943170+0000", 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000133225 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.943201+0000", 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1287e-05 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.385 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.947848+0000", 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0046467130000000002 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.947899+0000", 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1326000000000003e-05 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 66/33 ECSubWrite(tid=140, reqid=osd.4.0:140, at_version=66'67, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:01:57.386 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.728419+0000", 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5800808240000004, 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0082188199999999999, 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728419+0000", 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.387 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728419+0000", 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728419+0000", 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.65e-07 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728451+0000", 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1736000000000002e-05 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.388 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728469+0000", 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7819e-05 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728483+0000", 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4188999999999999e-05 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728942+0000", 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00045947400000000003 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.389 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728960+0000", 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7805e-05 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.736576+0000", 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0076158750000000002 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.736638+0000", 2024-09-16T04:01:57.390 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1556999999999994e-05 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 66/33 ECSubWrite(tid=141, reqid=osd.4.0:141, at_version=66'69, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.730367+0000", 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5781323890000003, 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011069918999999999, 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.391 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730367+0000", 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730367+0000", 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730369+0000", 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:01:57.392 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730408+0000", 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8977000000000003e-05 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730425+0000", 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7597000000000001e-05 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730435+0000", 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5619999999999993e-06 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730506+0000", 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0817999999999999e-05 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730515+0000", 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6360000000000006e-06 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.741355+0000", 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010840116 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.741437+0000", 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1710000000000002e-05 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 70/33 ECSubWrite(tid=248, reqid=client.4646.0:609, at_version=70'150, trim_to=0'0, roll_forward_to=69'148))", 2024-09-16T04:01:57.395 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:55.903709+0000", 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.4047905700000001, 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0044929510000000002, 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.903709+0000", 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.396 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.903709+0000", 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.903709+0000", 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3700000000000001e-07 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.903741+0000", 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1208999999999999e-05 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.397 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.903768+0000", 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7083000000000001e-05 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.903781+0000", 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3514e-05 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.904003+0000", 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022154400000000001 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.904016+0000", 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3429e-05 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.908156+0000", 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0041393649999999999 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.908202+0000", 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6369999999999998e-05 2024-09-16T04:01:57.399 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 70/33 ECSubWrite(tid=249, reqid=client.4646.0:610, at_version=70'151, trim_to=0'0, roll_forward_to=69'148))", 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:55.906716+0000", 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.4017839350000001, 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0041156420000000001, 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.400 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.906716+0000", 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.906716+0000", 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.906717+0000", 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2619999999999999e-06 2024-09-16T04:01:57.401 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.906748+0000", 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1279999999999999e-05 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.906767+0000", 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8374999999999999e-05 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.906781+0000", 2024-09-16T04:01:57.402 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4914e-05 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.907104+0000", 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00032226200000000003 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.907113+0000", 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4569999999999994e-06 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.403 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.910789+0000", 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036758419999999999 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:55.910831+0000", 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2249999999999997e-05 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:01:57.404 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:01:57.406 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:01:57.406 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:57.404+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:57.507 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:57.501+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:57.558 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:01:57.608 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:57.605+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:57.709 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:01:57.705+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:01:57.728 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:57.728 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:01:57.728 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:01:57.728 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:01:57.728 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:57.742 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:01:57.810 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:01:57.805+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:01:57.828 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:01:57.869 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 44/33 ECSubWrite(tid=82, reqid=client.4646.0:352, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.928412+0000", 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.937914473999999, 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0050467949999999998, 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.870 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928412+0000", 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928412+0000", 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928413+0000", 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0499999999999999e-06 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928470+0000", 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7218e-05 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928506+0000", 2024-09-16T04:01:57.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5803000000000003e-05 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928531+0000", 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5142000000000001e-05 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929620+0000", 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010886800000000001 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929658+0000", 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7771999999999997e-05 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.933359+0000", 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003701307 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.933459+0000", 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9822999999999996e-05 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.874 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 51/33 ECSubWrite(tid=89, reqid=osd.4.0:89, at_version=51'33, trim_to=0'0, roll_forward_to=50'31))", 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.719637+0000", 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.146689937000001, 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0092214389999999997, 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.875 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719637+0000", 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719637+0000", 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719637+0000", 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.13e-07 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719695+0000", 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7408000000000003e-05 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719732+0000", 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7579000000000001e-05 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719757+0000", 2024-09-16T04:01:57.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4856999999999999e-05 2024-09-16T04:01:57.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719919+0000", 2024-09-16T04:01:57.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016154200000000001 2024-09-16T04:01:57.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.719936+0000", 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7595999999999999e-05 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.728769+0000", 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0088326029999999996 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.728858+0000", 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9240999999999995e-05 2024-09-16T04:01:57.879 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 51/33 ECSubWrite(tid=71, reqid=client.4646.0:371, at_version=51'49, trim_to=0'0, roll_forward_to=42'47))", 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.726849+0000", 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.139477866, 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046563330000000003, 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726849+0000", 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726849+0000", 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726849+0000", 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:01:57.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726904+0000", 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4698999999999999e-05 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726935+0000", 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1259e-05 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.726959+0000", 2024-09-16T04:01:57.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3856e-05 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727186+0000", 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022679699999999999 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727215+0000", 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.923e-05 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.731406+0000", 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004190839 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.731505+0000", 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9170000000000001e-05 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 54/33 ECSubWrite(tid=86, reqid=osd.14.0:86, at_version=54'55, trim_to=0'0, roll_forward_to=52'53))", 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.800790+0000", 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.065536875999999, 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045216520000000001, 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800790+0000", 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800790+0000", 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800792+0000", 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.967e-06 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800847+0000", 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5627999999999997e-05 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800874+0000", 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6659999999999999e-05 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.800894+0000", 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9939e-05 2024-09-16T04:01:57.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.888 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.888 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.801124+0000", 2024-09-16T04:01:57.888 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022965799999999999 2024-09-16T04:01:57.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.888 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.888 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.801160+0000", 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6075999999999998e-05 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.805209+0000", 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040491329999999999 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.889 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.805311+0000", 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010259099999999999 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 54/33 ECSubWrite(tid=101, reqid=osd.4.0:101, at_version=54'44, trim_to=0'0, roll_forward_to=52'42))", 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.802102+0000", 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.064224511999999, 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0047704360000000003, 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.890 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802102+0000", 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802102+0000", 2024-09-16T04:01:57.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802103+0000", 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802157+0000", 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4005999999999998e-05 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802190+0000", 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3009000000000002e-05 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802212+0000", 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1855999999999999e-05 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802392+0000", 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018071499999999999 2024-09-16T04:01:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802418+0000", 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5516000000000001e-05 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.806782+0000", 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0043635790000000002 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.806872+0000", 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0977999999999995e-05 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 54/33 ECSubWrite(tid=102, reqid=osd.4.0:102, at_version=54'46, trim_to=0'0, roll_forward_to=52'42))", 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.802262+0000", 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.064064871999999, 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053342440000000001, 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.895 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802262+0000", 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802262+0000", 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802263+0000", 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7600000000000006e-07 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802316+0000", 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2855999999999997e-05 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802342+0000", 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6166e-05 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.802365+0000", 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3272999999999999e-05 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.804138+0000", 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017730090000000001 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.804163+0000", 2024-09-16T04:01:57.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5199e-05 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.807527+0000", 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033641840000000001 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.807596+0000", 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8580999999999999e-05 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 54/33 ECSubWrite(tid=104, reqid=client.4646.0:423, at_version=54'47, trim_to=0'0, roll_forward_to=54'46))", 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:39.832982+0000", 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.033344729, 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004648386, 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.832982+0000", 2024-09-16T04:01:57.900 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.832982+0000", 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.832984+0000", 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.68e-06 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834372+0000", 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013879929999999999 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834403+0000", 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834419+0000", 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5766000000000001e-05 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834538+0000", 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011849399999999999 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.834554+0000", 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-05 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.837545+0000", 2024-09-16T04:01:57.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029916029999999998 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:39.837630+0000", 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4890999999999997e-05 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.8s3 60/33 ECSubWrite(tid=12, reqid=client.4646.0:458, at_version=60'10, trim_to=0'0, roll_forward_to=41'8))", 2024-09-16T04:01:57.904 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.784656+0000", 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.081670387000001, 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0059195790000000003, 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.784656+0000", 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.905 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.784656+0000", 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.784657+0000", 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2559999999999999e-06 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.785140+0000", 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00048282899999999999 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.785419+0000", 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00027847899999999999 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.785463+0000", 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3775e-05 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.907 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.785692+0000", 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000229278 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.785725+0000", 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3231000000000001e-05 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.790512+0000", 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0047873270000000001 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.790576+0000", 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3404000000000005e-05 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 65/33 ECSubWrite(tid=98, reqid=client.4646.0:512, at_version=65'62, trim_to=0'0, roll_forward_to=63'60))", 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:50.901321+0000", 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.9650059080000002, 2024-09-16T04:01:57.909 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00424319, 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.901321+0000", 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.901321+0000", 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.901322+0000", 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8000000000000004e-07 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.902458+0000", 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011364770000000001 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.902503+0000", 2024-09-16T04:01:57.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5231000000000001e-05 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.902532+0000", 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8645000000000001e-05 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.902645+0000", 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000112862 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.912 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.902659+0000", 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3987e-05 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.905516+0000", 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0028569670000000002 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:50.905564+0000", 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8140999999999999e-05 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 66/33 ECSubWrite(tid=100, reqid=osd.14.0:100, at_version=66'64, trim_to=0'0, roll_forward_to=65'62))", 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.723508+0000", 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.1428188610000003, 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049707620000000001, 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723508+0000", 2024-09-16T04:01:57.914 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723508+0000", 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723508+0000", 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3e-07 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.915 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723543+0000", 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4999000000000002e-05 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723563+0000", 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9405000000000001e-05 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723578+0000", 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5293000000000001e-05 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723752+0000", 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017408199999999999 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723781+0000", 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9312e-05 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728375+0000", 2024-09-16T04:01:57.917 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004593626 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728478+0000", 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103415 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 66/33 ECSubWrite(tid=101, reqid=osd.14.0:101, at_version=66'66, trim_to=0'0, roll_forward_to=65'62))", 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.724926+0000", 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.1414007430000002, 2024-09-16T04:01:57.918 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0094705459999999998, 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724926+0000", 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724926+0000", 2024-09-16T04:01:57.919 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724928+0000", 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9300000000000002e-06 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724980+0000", 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2463999999999998e-05 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.725014+0000", 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4004999999999997e-05 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.725037+0000", 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2523000000000001e-05 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.725751+0000", 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00071411499999999998 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.921 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.725777+0000", 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6329000000000002e-05 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.734288+0000", 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.008511111 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.734396+0000", 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108069 2024-09-16T04:01:57.922 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/33 ECSubWrite(tid=140, reqid=osd.4.0:140, at_version=66'67, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.730775+0000", 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.1355519029999996, 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0059264979999999997, 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.923 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730775+0000", 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730775+0000", 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730776+0000", 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1400000000000001e-06 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730837+0000", 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1214e-05 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730858+0000", 2024-09-16T04:01:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1019e-05 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730880+0000", 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1963e-05 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731328+0000", 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00044770399999999999 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.926 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731353+0000", 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5715e-05 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.736603+0000", 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052500740000000004 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.736701+0000", 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7669000000000003e-05 2024-09-16T04:01:57.927 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/33 ECSubWrite(tid=141, reqid=osd.4.0:141, at_version=66'69, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.730939+0000", 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.1353879960000004, 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010155496, 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.928 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730939+0000", 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730939+0000", 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730940+0000", 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1960000000000001e-06 2024-09-16T04:01:57.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730998+0000", 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8137000000000003e-05 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731025+0000", 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6749000000000001e-05 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731044+0000", 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9306000000000001e-05 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733535+0000", 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0024914049999999999 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733563+0000", 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7378000000000001e-05 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740991+0000", 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0074285530000000001 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.741094+0000", 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000102772 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.932 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/33 ECSubWrite(tid=103, reqid=client.4646.0:555, at_version=67'68, trim_to=0'0, roll_forward_to=66'66))", 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:52.753134+0000", 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.1131922459999997, 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057082859999999999, 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.753134+0000", 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.753134+0000", 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.753135+0000", 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2600000000000002e-07 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.753602+0000", 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00046722000000000002 2024-09-16T04:01:57.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.753634+0000", 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2119000000000002e-05 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.753653+0000", 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8955999999999999e-05 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.753822+0000", 2024-09-16T04:01:57.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000169266 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.753850+0000", 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7824999999999999e-05 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.758770+0000", 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049194269999999997 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.758843+0000", 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2947000000000005e-05 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/33 ECSubWrite(tid=104, reqid=client.4646.0:556, at_version=67'69, trim_to=0'0, roll_forward_to=66'66))", 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:52.760100+0000", 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.1062267139999999, 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048315370000000003, 2024-09-16T04:01:57.937 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.760100+0000", 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.760100+0000", 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.760102+0000", 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.796e-06 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.760449+0000", 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00034716599999999999 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.760478+0000", 2024-09-16T04:01:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9518000000000001e-05 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.760494+0000", 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5665999999999999e-05 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.760616+0000", 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000121858 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.760631+0000", 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5427e-05 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.764844+0000", 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042125169999999998 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.764931+0000", 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7589000000000005e-05 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 67/33 ECSubWrite(tid=106, reqid=client.4646.0:560, at_version=67'17, trim_to=0'0, roll_forward_to=56'15))", 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:52.763941+0000", 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.1023852749999996, 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0081121309999999999, 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.763941+0000", 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.763941+0000", 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.763943+0000", 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9700000000000002e-06 2024-09-16T04:01:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.763998+0000", 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5188000000000001e-05 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.764036+0000", 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7115e-05 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.764063+0000", 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7455000000000001e-05 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.764238+0000", 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017454999999999999 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.764265+0000", 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7259000000000001e-05 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.771999+0000", 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0077340059999999999 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.772053+0000", 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4588e-05 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/33 ECSubWrite(tid=107, reqid=client.4646.0:558, at_version=67'71, trim_to=0'0, roll_forward_to=67'68))", 2024-09-16T04:01:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:52.767042+0000", 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.099284784, 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0094360739999999992, 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767042+0000", 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767042+0000", 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767044+0000", 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1270000000000001e-06 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767084+0000", 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0250999999999998e-05 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767116+0000", 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1559000000000001e-05 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767139+0000", 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3538999999999999e-05 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767288+0000", 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014833000000000001 2024-09-16T04:01:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767312+0000", 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3938999999999999e-05 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.776377+0000", 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0090654770000000006 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.776478+0000", 2024-09-16T04:01:57.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000100852 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 67/33 ECSubWrite(tid=108, reqid=client.4646.0:561, at_version=67'18, trim_to=0'0, roll_forward_to=56'15))", 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:52.767198+0000", 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.099128221, 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0093159030000000004, 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767198+0000", 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767198+0000", 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767199+0000", 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:01:57.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767253+0000", 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3195000000000001e-05 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767278+0000", 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5012e-05 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.953 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767295+0000", 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7762999999999999e-05 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767450+0000", 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015492300000000001 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.767476+0000", 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5819000000000001e-05 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.954 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.776446+0000", 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0089699110000000006 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.776514+0000", 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8159999999999998e-05 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 67/33 ECSubWrite(tid=109, reqid=client.4646.0:562, at_version=67'19, trim_to=0'0, roll_forward_to=56'15))", 2024-09-16T04:01:57.955 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:52.769519+0000", 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.0968072859999998, 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070400230000000003, 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.769519+0000", 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.769519+0000", 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.769521+0000", 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.313e-06 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.769550+0000", 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9499e-05 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.769576+0000", 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6387999999999999e-05 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.769598+0000", 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1256000000000001e-05 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.769718+0000", 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120195 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.769732+0000", 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4463e-05 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.776520+0000", 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0067881039999999997 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.776559+0000", 2024-09-16T04:01:57.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8804999999999999e-05 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.4s3 67/33 ECSubWrite(tid=57, reqid=client.4646.0:566, at_version=67'35, trim_to=0'0, roll_forward_to=66'33))", 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:52.774993+0000", 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.0913331599999996, 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044023090000000001, 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:01:57.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.774993+0000", 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.774993+0000", 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.774994+0000", 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5400000000000002e-07 2024-09-16T04:01:57.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.775623+0000", 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062870999999999999 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.775657+0000", 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3964000000000002e-05 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.775682+0000", 2024-09-16T04:01:57.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5456000000000002e-05 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.775879+0000", 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000197092 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.775910+0000", 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0654999999999997e-05 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:01:57.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.779294+0000", 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033845989999999999 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:52.779396+0000", 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010137900000000001 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:01:57.964 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:01:57.965 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:01:57.968 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T04:01:57.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:01:57.969 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:57.961+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:57.969 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T04:01:57.969 INFO:tasks.rados.rados.0.smithi188.stdout:243: done (3 left) 2024-09-16T04:01:57.969 INFO:tasks.rados.rados.0.smithi188.stdout:244: done (2 left) 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:245: done (1 left) 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:246: done (0 left) 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:247: read oid 22 snap -1 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:247: expect (ObjNum 82 snap 10 seq_num 82) 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:248: read oid 41 snap -1 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:248: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:249: read oid 40 snap -1 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:249: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:250: delete oid 14 current snap is 26 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:250: done (3 left) 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 37 current snap is 26 2024-09-16T04:01:57.970 INFO:tasks.rados.rados.0.smithi188.stdout:251: seq_num 111 ranges {4751360=704512,5455872=655360,6111232=507904} 2024-09-16T04:01:57.989 INFO:tasks.rados.rados.0.smithi188.stdout:251: writing smithi18831615-37 from 4751360 to 5455872 tid 1 2024-09-16T04:01:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:01:57.990 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:01:57.994 INFO:tasks.rados.rados.0.smithi188.stdout:251: writing smithi18831615-37 from 5455872 to 6111232 tid 2 2024-09-16T04:01:57.999 INFO:tasks.rados.rados.0.smithi188.stdout:251: writing smithi18831615-37 from 6111232 to 6619136 tid 3 2024-09-16T04:01:57.999 INFO:tasks.rados.rados.0.smithi188.stdout:252: copy_from oid 11 from oid 7 current snap is 26 2024-09-16T04:01:57.999 INFO:tasks.rados.rados.0.smithi188.stdout:249: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:01:58.001 INFO:tasks.rados.rados.0.smithi188.stdout:247: expect (ObjNum 82 snap 10 seq_num 82) 2024-09-16T04:01:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:01:58.016 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:01:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:01:58.037 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:01:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:01:58.059 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:01:58.066 INFO:tasks.rados.rados.0.smithi188.stdout:248: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:01:58.066 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:58.061+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:58.075 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:01:58.076 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:01:58.084 INFO:tasks.rados.rados.0.smithi188.stdout:251: finishing write tid 1 to smithi18831615-37 2024-09-16T04:01:58.084 INFO:tasks.rados.rados.0.smithi188.stdout:251: finishing write tid 2 to smithi18831615-37 2024-09-16T04:01:58.084 INFO:tasks.rados.rados.0.smithi188.stdout:251: finishing write tid 3 to smithi18831615-37 2024-09-16T04:01:58.084 INFO:tasks.rados.rados.0.smithi188.stdout:251: finishing write tid 4 to smithi18831615-37 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:251: finishing write tid 5 to smithi18831615-37 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:251: oid 37 updating version 0 to 157 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:251: oid 37 version 157 is already newer than 154 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:251: oid 37 version 157 is already newer than 156 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:251: oid 37 version 157 is already newer than 155 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 37 v 157 (ObjNum 111 snap 26 seq_num 111) dirty exists 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:251: left oid 37 (ObjNum 111 snap 26 seq_num 111) 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:252: finishing copy_from racing read to smithi18831615-11 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:252: finishing copy_from to smithi18831615-11 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 11 v 51 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:247: done (4 left) 2024-09-16T04:01:58.085 INFO:tasks.rados.rados.0.smithi188.stdout:248: done (3 left) 2024-09-16T04:01:58.086 INFO:tasks.rados.rados.0.smithi188.stdout:249: done (2 left) 2024-09-16T04:01:58.086 INFO:tasks.rados.rados.0.smithi188.stdout:251: done (1 left) 2024-09-16T04:01:58.086 INFO:tasks.rados.rados.0.smithi188.stdout:252: done (0 left) 2024-09-16T04:01:58.086 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 35 current snap is 26 2024-09-16T04:01:58.086 INFO:tasks.rados.rados.0.smithi188.stdout:253: seq_num 112 ranges {3063808=720896,3784704=802816,4587520=360448} 2024-09-16T04:01:58.095 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:01:58.096 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:01:58.121 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:01:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:01:58.127 INFO:tasks.rados.rados.0.smithi188.stdout:253: writing smithi18831615-35 from 3063808 to 3784704 tid 1 2024-09-16T04:01:58.136 INFO:tasks.rados.rados.0.smithi188.stdout:253: writing smithi18831615-35 from 3784704 to 4587520 tid 2 2024-09-16T04:01:58.140 INFO:tasks.rados.rados.0.smithi188.stdout:253: writing smithi18831615-35 from 4587520 to 4947968 tid 3 2024-09-16T04:01:58.140 INFO:tasks.rados.rados.0.smithi188.stdout:254: read oid 29 snap -1 2024-09-16T04:01:58.140 INFO:tasks.rados.rados.0.smithi188.stdout:254: expect (ObjNum 104 snap 22 seq_num 104) 2024-09-16T04:01:58.141 INFO:tasks.rados.rados.0.smithi188.stdout:255: delete oid 13 current snap is 26 2024-09-16T04:01:58.141 INFO:tasks.rados.rados.0.smithi188.stdout:253: finishing write tid 1 to smithi18831615-35 2024-09-16T04:01:58.146 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:01:58.146 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:01:58.148 INFO:tasks.rados.rados.0.smithi188.stdout:255: done (2 left) 2024-09-16T04:01:58.148 INFO:tasks.rados.rados.0.smithi188.stdout:256: copy_from oid 5 from oid 13 current snap is 26 2024-09-16T04:01:58.148 INFO:tasks.rados.rados.0.smithi188.stdout:257: read oid 16 snap 21 2024-09-16T04:01:58.148 INFO:tasks.rados.rados.0.smithi188.stdout:257: expect (ObjNum 80 snap 10 seq_num 80) 2024-09-16T04:01:58.148 INFO:tasks.rados.rados.0.smithi188.stdout:258: read oid 47 snap -1 2024-09-16T04:01:58.149 INFO:tasks.rados.rados.0.smithi188.stdout:258: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:01:58.149 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 37 current snap is 26 2024-09-16T04:01:58.149 INFO:tasks.rados.rados.0.smithi188.stdout:259: seq_num 114 ranges {6619136=753664,7372800=688128,8060928=475136} 2024-09-16T04:01:58.150 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:58.150 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:58.163 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:01:58.168 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:58.161+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:01:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:01:58.190 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:01:58.191 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:01:58.193 INFO:tasks.rados.rados.0.smithi188.stdout:259: writing smithi18831615-37 from 6619136 to 7372800 tid 1 2024-09-16T04:01:58.197 INFO:tasks.rados.rados.0.smithi188.stdout:259: writing smithi18831615-37 from 7372800 to 8060928 tid 2 2024-09-16T04:01:58.202 INFO:tasks.rados.rados.0.smithi188.stdout:259: writing smithi18831615-37 from 8060928 to 8536064 tid 3 2024-09-16T04:01:58.202 INFO:tasks.rados.rados.0.smithi188.stdout:260: read oid 43 snap 18 2024-09-16T04:01:58.202 INFO:tasks.rados.rados.0.smithi188.stdout:260: expect (ObjNum 60 snap 1 seq_num 60) 2024-09-16T04:01:58.202 INFO:tasks.rados.rados.0.smithi188.stdout:261: rollback oid 44 current snap is 26 2024-09-16T04:01:58.202 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 44 to 18 2024-09-16T04:01:58.202 INFO:tasks.rados.rados.0.smithi188.stdout:253: finishing write tid 2 to smithi18831615-35 2024-09-16T04:01:58.202 INFO:tasks.rados.rados.0.smithi188.stdout:256: finishing copy_from to smithi18831615-5 2024-09-16T04:01:58.202 INFO:tasks.rados.rados.0.smithi188.stdout:256: got expected ENOENT (src dne) 2024-09-16T04:01:58.203 INFO:tasks.rados.rados.0.smithi188.stdout:256: finishing copy_from racing read to smithi18831615-5 2024-09-16T04:01:58.203 INFO:tasks.rados.rados.0.smithi188.stdout:253: finishing write tid 3 to smithi18831615-35 2024-09-16T04:01:58.203 INFO:tasks.rados.rados.0.smithi188.stdout:256: done (7 left) 2024-09-16T04:01:58.203 INFO:tasks.rados.rados.0.smithi188.stdout:262: read oid 42 snap -1 2024-09-16T04:01:58.203 INFO:tasks.rados.rados.0.smithi188.stdout:262: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:58.203 INFO:tasks.rados.rados.0.smithi188.stdout:263: read oid 12 snap -1 2024-09-16T04:01:58.203 INFO:tasks.rados.rados.0.smithi188.stdout:263: expect (ObjNum 101 snap 21 seq_num 101) 2024-09-16T04:01:58.203 INFO:tasks.rados.rados.0.smithi188.stdout:264: snap_remove snap 24 2024-09-16T04:01:58.221 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:01:58.221 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:01:58.243 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:01:58.243 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:01:58.261 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:01:58.261 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:01:58.268 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:01:58.261+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:01:58.280 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:01:58.280 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:01:58.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:01:58.280+0000 7f7bbf7fe640 0 -- 172.21.15.37:0/1247093036 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7f7bc0223500 msgr2=0x7f7bc02258f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:01:58.295 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:58.295 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:58.302 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:01:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:01:58.310 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:01:58.322 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:01:58.322 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:01:58.369 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:58.361+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:58.435 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:58.436 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:58.450 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:01:58.470 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:01:58.469+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:01:58.571 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:58.568+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:58.623 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:58.623 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:58.638 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:01:58.682 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:58.677+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:58.759 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:58.759 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:58.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:01:58.775 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:58.769+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:58.874 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:58.869+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:58.922 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T04:01:58.923 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-09-16T04:01:58.928 INFO:tasks.rados.rados.0.smithi188.stdout:264: done (9 left) 2024-09-16T04:01:58.928 INFO:tasks.rados.rados.0.smithi188.stdout:265: delete oid 49 current snap is 26 2024-09-16T04:01:58.928 INFO:tasks.rados.rados.0.smithi188.stdout:253: finishing write tid 4 to smithi18831615-35 2024-09-16T04:01:58.928 INFO:tasks.rados.rados.0.smithi188.stdout:254: expect (ObjNum 104 snap 22 seq_num 104) 2024-09-16T04:01:58.956 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:58.956 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:58.957 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T04:01:58.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:58.970 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:01:59.008 INFO:tasks.rados.rados.0.smithi188.stdout:253: finishing write tid 5 to smithi18831615-35 2024-09-16T04:01:59.008 INFO:tasks.rados.rados.0.smithi188.stdout:253: oid 35 updating version 0 to 55 2024-09-16T04:01:59.008 INFO:tasks.rados.rados.0.smithi188.stdout:253: oid 35 version 55 is already newer than 54 2024-09-16T04:01:59.008 INFO:tasks.rados.rados.0.smithi188.stdout:253: oid 35 version 55 is already newer than 52 2024-09-16T04:01:59.008 INFO:tasks.rados.rados.0.smithi188.stdout:253: oid 35 version 55 is already newer than 53 2024-09-16T04:01:59.008 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 55 (ObjNum 112 snap 26 seq_num 112) dirty exists 2024-09-16T04:01:59.008 INFO:tasks.rados.rados.0.smithi188.stdout:253: left oid 35 (ObjNum 112 snap 26 seq_num 112) 2024-09-16T04:01:59.008 INFO:tasks.rados.rados.0.smithi188.stdout:258: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:01:59.014 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:59.009+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:59.027 INFO:tasks.rados.rados.0.smithi188.stdout:257: expect (ObjNum 80 snap 10 seq_num 80) 2024-09-16T04:01:59.076 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:01:59.073+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:01:59.114 INFO:tasks.rados.rados.0.smithi188.stdout:259: finishing write tid 1 to smithi18831615-37 2024-09-16T04:01:59.114 INFO:tasks.rados.rados.0.smithi188.stdout:259: finishing write tid 2 to smithi18831615-37 2024-09-16T04:01:59.114 INFO:tasks.rados.rados.0.smithi188.stdout:259: finishing write tid 3 to smithi18831615-37 2024-09-16T04:01:59.115 INFO:tasks.rados.rados.0.smithi188.stdout:259: finishing write tid 4 to smithi18831615-37 2024-09-16T04:01:59.115 INFO:tasks.rados.rados.0.smithi188.stdout:261: finishing rollback tid 0 to smithi18831615-44 2024-09-16T04:01:59.115 INFO:tasks.rados.rados.0.smithi188.stdout:262: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:01:59.132 INFO:tasks.rados.rados.0.smithi188.stdout:259: finishing write tid 5 to smithi18831615-37 2024-09-16T04:01:59.132 INFO:tasks.rados.rados.0.smithi188.stdout:259: oid 37 updating version 0 to 161 2024-09-16T04:01:59.132 INFO:tasks.rados.rados.0.smithi188.stdout:259: oid 37 version 161 is already newer than 159 2024-09-16T04:01:59.132 INFO:tasks.rados.rados.0.smithi188.stdout:259: oid 37 version 161 is already newer than 160 2024-09-16T04:01:59.132 INFO:tasks.rados.rados.0.smithi188.stdout:259: oid 37 updating version 161 to 162 2024-09-16T04:01:59.132 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 37 v 162 (ObjNum 114 snap 26 seq_num 114) dirty exists 2024-09-16T04:01:59.132 INFO:tasks.rados.rados.0.smithi188.stdout:259: left oid 37 (ObjNum 114 snap 26 seq_num 114) 2024-09-16T04:01:59.132 INFO:tasks.rados.rados.0.smithi188.stdout:261: finishing rollback tid 1 to smithi18831615-44 2024-09-16T04:01:59.133 INFO:tasks.rados.rados.0.smithi188.stdout:261: finishing rollback tid 2 to smithi18831615-44 2024-09-16T04:01:59.133 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 44 v 58 (ObjNum 69 snap 2 seq_num 69) dirty exists 2024-09-16T04:01:59.133 INFO:tasks.rados.rados.0.smithi188.stdout:260: expect (ObjNum 60 snap 1 seq_num 60) 2024-09-16T04:01:59.135 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:59.136 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:59.150 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:01:59.177 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:01:59.173+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:01:59.189 INFO:tasks.rados.rados.0.smithi188.stdout:263: expect (ObjNum 101 snap 21 seq_num 101) 2024-09-16T04:01:59.270 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:59.271 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:59.278 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:59.276+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:59.283 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:01:59.304 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:59.305 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:59.306 INFO:tasks.rados.rados.0.smithi188.stdout:253: done (9 left) 2024-09-16T04:01:59.306 INFO:tasks.rados.rados.0.smithi188.stdout:254: done (8 left) 2024-09-16T04:01:59.306 INFO:tasks.rados.rados.0.smithi188.stdout:257: done (7 left) 2024-09-16T04:01:59.306 INFO:tasks.rados.rados.0.smithi188.stdout:258: done (6 left) 2024-09-16T04:01:59.306 INFO:tasks.rados.rados.0.smithi188.stdout:259: done (5 left) 2024-09-16T04:01:59.306 INFO:tasks.rados.rados.0.smithi188.stdout:260: done (4 left) 2024-09-16T04:01:59.306 INFO:tasks.rados.rados.0.smithi188.stdout:261: done (3 left) 2024-09-16T04:01:59.306 INFO:tasks.rados.rados.0.smithi188.stdout:262: done (2 left) 2024-09-16T04:01:59.307 INFO:tasks.rados.rados.0.smithi188.stdout:263: done (1 left) 2024-09-16T04:01:59.307 INFO:tasks.rados.rados.0.smithi188.stdout:265: done (0 left) 2024-09-16T04:01:59.307 INFO:tasks.rados.rados.0.smithi188.stdout:266: copy_from oid 25 from oid 31 current snap is 26 2024-09-16T04:01:59.307 INFO:tasks.rados.rados.0.smithi188.stdout:267: snap_create 2024-09-16T04:01:59.323 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-16T04:01:59.056219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93372979,"num_objects":95,"num_object_clones":42,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":3,"num_read":456,"num_read_kb":100884,"num_write":775,"num_write_kb":103878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":653904,"kb_used_data":175648,"kb_used_omap":24,"kb_used_meta":477863,"kb_avail":1509295536,"statfs":{"total":1546188226560,"available":1545518628864,"internally_reserved":0,"allocated":179863552,"data_stored":232426355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":489331793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15597568,"num_objects":9,"num_object_clones":9,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":-2,"num_read":34,"num_read_kb":20271,"num_write":34,"num_write_kb":15232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001093"},"pg_stats":[{"pgid":"3.f","version":"69'47","reported_seq":222,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901578+0000","last_change":"2024-09-16T04:01:54.932899+0000","last_active":"2024-09-16T04:01:55.901578+0000","last_peered":"2024-09-16T04:01:55.901578+0000","last_clean":"2024-09-16T04:01:55.901578+0000","last_became_active":"2024-09-16T04:01:19.845422+0000","last_became_peered":"2024-09-16T04:01:19.845422+0000","last_unstale":"2024-09-16T04:01:55.901578+0000","last_undegraded":"2024-09-16T04:01:55.901578+0000","last_fullsized":"2024-09-16T04:01:55.901578+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:31.239242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036964929000000001,"stat_sum":{"num_bytes":8847360,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":26,"num_read_kb":6985,"num_write":46,"num_write_kb":9168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.e","version":"43'6","reported_seq":186,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.923474+0000","last_change":"2024-09-16T04:01:54.908308+0000","last_active":"2024-09-16T04:01:57.923474+0000","last_peered":"2024-09-16T04:01:57.923474+0000","last_clean":"2024-09-16T04:01:57.923474+0000","last_became_active":"2024-09-16T04:01:20.266249+0000","last_became_peered":"2024-09-16T04:01:20.266249+0000","last_unstale":"2024-09-16T04:01:57.923474+0000","last_undegraded":"2024-09-16T04:01:57.923474+0000","last_fullsized":"2024-09-16T04:01:57.923474+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:42.438971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042994230000000001,"stat_sum":{"num_bytes":1064960,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3317,"num_write":6,"num_write_kb":1440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.d","version":"69'40","reported_seq":219,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.953810+0000","last_change":"2024-09-16T04:01:54.912961+0000","last_active":"2024-09-16T04:01:56.953810+0000","last_peered":"2024-09-16T04:01:56.953810+0000","last_clean":"2024-09-16T04:01:56.953810+0000","last_became_active":"2024-09-16T04:01:19.845009+0000","last_became_peered":"2024-09-16T04:01:19.845009+0000","last_unstale":"2024-09-16T04:01:56.953810+0000","last_undegraded":"2024-09-16T04:01:56.953810+0000","last_fullsized":"2024-09-16T04:01:56.953810+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.998546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023860605,"stat_sum":{"num_bytes":9322496,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":15587,"num_write":40,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.c","version":"70'152","reported_seq":322,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.914582+0000","last_change":"2024-09-16T04:01:54.937443+0000","last_active":"2024-09-16T04:01:55.914582+0000","last_peered":"2024-09-16T04:01:55.914582+0000","last_clean":"2024-09-16T04:01:55.914582+0000","last_became_active":"2024-09-16T04:01:20.259789+0000","last_became_peered":"2024-09-16T04:01:20.259789+0000","last_unstale":"2024-09-16T04:01:55.914582+0000","last_undegraded":"2024-09-16T04:01:55.914582+0000","last_fullsized":"2024-09-16T04:01:55.914582+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:39.754282+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.041360013000000001,"stat_sum":{"num_bytes":17055744,"num_objects":18,"num_object_clones":11,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":48,"num_read_kb":16909,"num_write":134,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.b","version":"69'82","reported_seq":255,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927364+0000","last_change":"2024-09-16T04:01:54.929640+0000","last_active":"2024-09-16T04:01:55.927364+0000","last_peered":"2024-09-16T04:01:55.927364+0000","last_clean":"2024-09-16T04:01:55.927364+0000","last_became_active":"2024-09-16T04:01:20.268548+0000","last_became_peered":"2024-09-16T04:01:20.268548+0000","last_unstale":"2024-09-16T04:01:55.927364+0000","last_undegraded":"2024-09-16T04:01:55.927364+0000","last_fullsized":"2024-09-16T04:01:55.927364+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:21.113712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033503993000000003,"stat_sum":{"num_bytes":12779520,"num_objects":11,"num_object_clones":5,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":40,"num_read_kb":5152,"num_write":64,"num_write_kb":13392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.a","version":"42'10","reported_seq":182,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.274216+0000","last_change":"2024-09-16T04:01:54.891010+0000","last_active":"2024-09-16T04:01:57.274216+0000","last_peered":"2024-09-16T04:01:57.274216+0000","last_clean":"2024-09-16T04:01:57.274216+0000","last_became_active":"2024-09-16T04:01:20.268001+0000","last_became_peered":"2024-09-16T04:01:20.268001+0000","last_unstale":"2024-09-16T04:01:57.274216+0000","last_undegraded":"2024-09-16T04:01:57.274216+0000","last_fullsized":"2024-09-16T04:01:57.274216+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.375530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022714649999999999,"stat_sum":{"num_bytes":704512,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":6196,"num_write":4,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.9","version":"69'21","reported_seq":197,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.943328+0000","last_change":"2024-09-16T04:01:54.917485+0000","last_active":"2024-09-16T04:01:57.943328+0000","last_peered":"2024-09-16T04:01:57.943328+0000","last_clean":"2024-09-16T04:01:57.943328+0000","last_became_active":"2024-09-16T04:01:20.153728+0000","last_became_peered":"2024-09-16T04:01:20.153728+0000","last_unstale":"2024-09-16T04:01:57.943328+0000","last_undegraded":"2024-09-16T04:01:57.943328+0000","last_fullsized":"2024-09-16T04:01:57.943328+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.150743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018434576000000001,"stat_sum":{"num_bytes":2441216,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7157,"num_write":10,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.8","version":"69'15","reported_seq":174,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901158+0000","last_change":"2024-09-16T04:01:54.928232+0000","last_active":"2024-09-16T04:01:55.901158+0000","last_peered":"2024-09-16T04:01:55.901158+0000","last_clean":"2024-09-16T04:01:55.901158+0000","last_became_active":"2024-09-16T04:01:20.259482+0000","last_became_peered":"2024-09-16T04:01:20.259482+0000","last_unstale":"2024-09-16T04:01:55.901158+0000","last_undegraded":"2024-09-16T04:01:55.901158+0000","last_fullsized":"2024-09-16T04:01:55.901158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.552136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017567564000000001,"stat_sum":{"num_bytes":3063808,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":298,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931845+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:56.931845+0000","last_peered":"2024-09-16T04:01:56.931845+0000","last_clean":"2024-09-16T04:01:56.931845+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:56.931845+0000","last_undegraded":"2024-09-16T04:01:56.931845+0000","last_fullsized":"2024-09-16T04:01:56.931845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":96,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901414+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:55.901414+0000","last_peered":"2024-09-16T04:01:55.901414+0000","last_clean":"2024-09-16T04:01:55.901414+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:55.901414+0000","last_undegraded":"2024-09-16T04:01:55.901414+0000","last_fullsized":"2024-09-16T04:01:55.901414+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"65'27","reported_seq":196,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.932186+0000","last_change":"2024-09-16T04:01:54.895830+0000","last_active":"2024-09-16T04:01:56.932186+0000","last_peered":"2024-09-16T04:01:56.932186+0000","last_clean":"2024-09-16T04:01:56.932186+0000","last_became_active":"2024-09-16T04:01:20.258661+0000","last_became_peered":"2024-09-16T04:01:20.258661+0000","last_unstale":"2024-09-16T04:01:56.932186+0000","last_undegraded":"2024-09-16T04:01:56.932186+0000","last_fullsized":"2024-09-16T04:01:56.932186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.280788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001177881,"stat_sum":{"num_bytes":3997696,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":6870,"num_write":18,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":99,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.273943+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:57.273943+0000","last_peered":"2024-09-16T04:01:57.273943+0000","last_clean":"2024-09-16T04:01:57.273943+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:57.273943+0000","last_undegraded":"2024-09-16T04:01:57.273943+0000","last_fullsized":"2024-09-16T04:01:57.273943+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'23","reported_seq":183,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927133+0000","last_change":"2024-09-16T04:01:54.912066+0000","last_active":"2024-09-16T04:01:55.927133+0000","last_peered":"2024-09-16T04:01:55.927133+0000","last_clean":"2024-09-16T04:01:55.927133+0000","last_became_active":"2024-09-16T04:01:20.269891+0000","last_became_peered":"2024-09-16T04:01:20.269891+0000","last_unstale":"2024-09-16T04:01:55.927133+0000","last_undegraded":"2024-09-16T04:01:55.927133+0000","last_fullsized":"2024-09-16T04:01:55.927133+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:33.263117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016000962000000001,"stat_sum":{"num_bytes":2621440,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":1540,"num_write":48,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":101,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931930+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:56.931930+0000","last_peered":"2024-09-16T04:01:56.931930+0000","last_clean":"2024-09-16T04:01:56.931930+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:56.931930+0000","last_undegraded":"2024-09-16T04:01:56.931930+0000","last_fullsized":"2024-09-16T04:01:56.931930+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"69'77","reported_seq":252,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.924464+0000","last_change":"2024-09-16T04:01:54.923402+0000","last_active":"2024-09-16T04:01:57.924464+0000","last_peered":"2024-09-16T04:01:57.924464+0000","last_clean":"2024-09-16T04:01:57.924464+0000","last_became_active":"2024-09-16T04:01:20.257526+0000","last_became_peered":"2024-09-16T04:01:20.257526+0000","last_unstale":"2024-09-16T04:01:57.924464+0000","last_undegraded":"2024-09-16T04:01:57.924464+0000","last_fullsized":"2024-09-16T04:01:57.924464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:21.464739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020041397999999998,"stat_sum":{"num_bytes":8192000,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":9951,"num_write":80,"num_write_kb":11696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":101,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.274219+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:57.274219+0000","last_peered":"2024-09-16T04:01:57.274219+0000","last_clean":"2024-09-16T04:01:57.274219+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:57.274219+0000","last_undegraded":"2024-09-16T04:01:57.274219+0000","last_fullsized":"2024-09-16T04:01:57.274219+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"70'46","reported_seq":216,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.932466+0000","last_change":"2024-09-16T04:01:54.924478+0000","last_active":"2024-09-16T04:01:56.932466+0000","last_peered":"2024-09-16T04:01:56.932466+0000","last_clean":"2024-09-16T04:01:56.932466+0000","last_became_active":"2024-09-16T04:01:19.840099+0000","last_became_peered":"2024-09-16T04:01:19.840099+0000","last_unstale":"2024-09-16T04:01:56.932466+0000","last_undegraded":"2024-09-16T04:01:56.932466+0000","last_fullsized":"2024-09-16T04:01:56.932466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:42.087321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029690392999999999,"stat_sum":{"num_bytes":7700480,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":18,"num_read_kb":12148,"num_write":60,"num_write_kb":7520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.5","version":"70'37","reported_seq":197,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.941837+0000","last_change":"2024-09-16T04:01:54.909344+0000","last_active":"2024-09-16T04:01:55.941837+0000","last_peered":"2024-09-16T04:01:55.941837+0000","last_clean":"2024-09-16T04:01:55.941837+0000","last_became_active":"2024-09-16T04:01:20.258954+0000","last_became_peered":"2024-09-16T04:01:20.258954+0000","last_unstale":"2024-09-16T04:01:55.941837+0000","last_undegraded":"2024-09-16T04:01:55.941837+0000","last_fullsized":"2024-09-16T04:01:55.941837+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.023122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019615982000000001,"stat_sum":{"num_bytes":7225344,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":18,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927159+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:55.927159+0000","last_peered":"2024-09-16T04:01:55.927159+0000","last_clean":"2024-09-16T04:01:55.927159+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:55.927159+0000","last_undegraded":"2024-09-16T04:01:55.927159+0000","last_fullsized":"2024-09-16T04:01:55.927159+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"69'42","reported_seq":219,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.906968+0000","last_change":"2024-09-16T04:01:54.923291+0000","last_active":"2024-09-16T04:01:55.906968+0000","last_peered":"2024-09-16T04:01:55.906968+0000","last_clean":"2024-09-16T04:01:55.906968+0000","last_became_active":"2024-09-16T04:01:20.154175+0000","last_became_peered":"2024-09-16T04:01:20.154175+0000","last_unstale":"2024-09-16T04:01:55.906968+0000","last_undegraded":"2024-09-16T04:01:55.906968+0000","last_fullsized":"2024-09-16T04:01:55.906968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:22.027112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025583108,"stat_sum":{"num_bytes":6062080,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":30,"num_read_kb":5723,"num_write":24,"num_write_kb":7280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:58.000142+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:58.000142+0000","last_peered":"2024-09-16T04:01:58.000142+0000","last_clean":"2024-09-16T04:01:58.000142+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:58.000142+0000","last_undegraded":"2024-09-16T04:01:58.000142+0000","last_fullsized":"2024-09-16T04:01:58.000142+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"36'7","reported_seq":171,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927164+0000","last_change":"2024-09-16T04:01:54.896709+0000","last_active":"2024-09-16T04:01:55.927164+0000","last_peered":"2024-09-16T04:01:55.927164+0000","last_clean":"2024-09-16T04:01:55.927164+0000","last_became_active":"2024-09-16T04:01:20.266522+0000","last_became_peered":"2024-09-16T04:01:20.266522+0000","last_unstale":"2024-09-16T04:01:55.927164+0000","last_undegraded":"2024-09-16T04:01:55.927164+0000","last_fullsized":"2024-09-16T04:01:55.927164+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.553251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000589103,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":724,"num_write":7,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":97,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.906581+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:55.906581+0000","last_peered":"2024-09-16T04:01:55.906581+0000","last_clean":"2024-09-16T04:01:55.906581+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:55.906581+0000","last_undegraded":"2024-09-16T04:01:55.906581+0000","last_fullsized":"2024-09-16T04:01:55.906581+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"36'2","reported_seq":168,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931956+0000","last_change":"2024-09-16T04:01:54.894048+0000","last_active":"2024-09-16T04:01:56.931956+0000","last_peered":"2024-09-16T04:01:56.931956+0000","last_clean":"2024-09-16T04:01:56.931956+0000","last_became_active":"2024-09-16T04:01:20.267489+0000","last_became_peered":"2024-09-16T04:01:20.267489+0000","last_unstale":"2024-09-16T04:01:56.931956+0000","last_undegraded":"2024-09-16T04:01:56.931956+0000","last_fullsized":"2024-09-16T04:01:56.931956+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:41.900963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078888799999999996,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2499,"num_write":2,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.897011+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:55.897011+0000","last_peered":"2024-09-16T04:01:55.897011+0000","last_clean":"2024-09-16T04:01:55.897011+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:55.897011+0000","last_undegraded":"2024-09-16T04:01:55.897011+0000","last_fullsized":"2024-09-16T04:01:55.897011+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92782592,"num_objects":91,"num_object_clones":42,"num_object_copies":637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":3,"num_read":320,"num_read_kb":100766,"num_write":570,"num_write_kb":100416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167907328,"data_stored":233668608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":634,"ondisk_log_size":634,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42040,"kb_used_data":10848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329800,"statfs":{"total":96636764160,"available":96593715200,"internally_reserved":0,"allocated":11108352,"data_stored":12998221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215120,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42444,"kb_used_data":11252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329396,"statfs":{"total":96636764160,"available":96593301504,"internally_reserved":0,"allocated":11522048,"data_stored":16832077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215121,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45676,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326164,"statfs":{"total":96636764160,"available":96589991936,"internally_reserved":0,"allocated":14831616,"data_stored":20842061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37424,"kb_used_data":10392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334416,"statfs":{"total":96636764160,"available":96598441984,"internally_reserved":0,"allocated":10641408,"data_stored":12002743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280529,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45240,"kb_used_data":14048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326600,"statfs":{"total":96636764160,"available":96590438400,"internally_reserved":0,"allocated":14385152,"data_stored":17157584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280527,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40888,"kb_used_data":13856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330952,"statfs":{"total":96636764160,"available":96594894848,"internally_reserved":0,"allocated":14188544,"data_stored":19269047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280527,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":5544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":5677056,"data_stored":5698999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280527,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":9636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":9867264,"data_stored":14097872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280527,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35620,"kb_used_data":8588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336220,"statfs":{"total":96636764160,"available":96600289280,"internally_reserved":0,"allocated":8794112,"data_stored":12793290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35500,"kb_used_data":8468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336340,"statfs":{"total":96636764160,"available":96600412160,"internally_reserved":0,"allocated":8671232,"data_stored":10634679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215120,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43284,"kb_used_data":12096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328556,"statfs":{"total":96636764160,"available":96592441344,"internally_reserved":0,"allocated":12386304,"data_stored":14766214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48864,"kb_used_data":17676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322976,"statfs":{"total":96636764160,"available":96586727424,"internally_reserved":0,"allocated":18100224,"data_stored":24260048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247824,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46292,"kb_used_data":10940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325548,"statfs":{"total":96636764160,"available":96589361152,"internally_reserved":0,"allocated":11202560,"data_stored":13232237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247824,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38044,"kb_used_data":6856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333796,"statfs":{"total":96636764160,"available":96597807104,"internally_reserved":0,"allocated":7020544,"data_stored":8314339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41884,"kb_used_data":10696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329956,"statfs":{"total":96636764160,"available":96593874944,"internally_reserved":0,"allocated":10952704,"data_stored":17194448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41456,"kb_used_data":10268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330384,"statfs":{"total":96636764160,"available":96594313216,"internally_reserved":0,"allocated":10514432,"data_stored":12332496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17584128,"data_stored":24752128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9682944,"data_stored":11468800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10846208,"data_stored":14147584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6041600,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13795328,"data_stored":16998400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9977856,"data_stored":16650240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14991360,"data_stored":22626304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9539584,"data_stored":11960320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13324288,"data_stored":19214336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4718592,"data_stored":5165056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":12152832,"data_stored":19853312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8892416,"data_stored":13553664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":7831552,"data_stored":12644352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":9678848,"data_stored":12148736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10588160,"data_stored":13680640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":8261632,"data_stored":11034624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:59.325 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:59.325 INFO:tasks.rados.rados.0.smithi188.stdout:266: finishing copy_from racing read to smithi18831615-25 2024-09-16T04:01:59.327 INFO:tasks.rados.rados.0.smithi188.stdout:266: finishing copy_from to smithi18831615-25 2024-09-16T04:01:59.327 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 25 v 27 (ObjNum 97 snap 19 seq_num 97) dirty exists 2024-09-16T04:01:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:59.413 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:01:59.412+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:01:59.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:01:59.480 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:59.477+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:59.550 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:59.551 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:59.565 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:01:59.581 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:59.577+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:01:59.675 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:01:59.676 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:01:59.682 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:01:59.680+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:01:59.683 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:59.683 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:59.694 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-16T04:01:59.056219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93372979,"num_objects":95,"num_object_clones":42,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":3,"num_read":456,"num_read_kb":100884,"num_write":775,"num_write_kb":103878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":653904,"kb_used_data":175648,"kb_used_omap":24,"kb_used_meta":477863,"kb_avail":1509295536,"statfs":{"total":1546188226560,"available":1545518628864,"internally_reserved":0,"allocated":179863552,"data_stored":232426355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":489331793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15597568,"num_objects":9,"num_object_clones":9,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":-2,"num_read":34,"num_read_kb":20271,"num_write":34,"num_write_kb":15232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001093"},"pg_stats":[{"pgid":"3.f","version":"69'47","reported_seq":222,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901578+0000","last_change":"2024-09-16T04:01:54.932899+0000","last_active":"2024-09-16T04:01:55.901578+0000","last_peered":"2024-09-16T04:01:55.901578+0000","last_clean":"2024-09-16T04:01:55.901578+0000","last_became_active":"2024-09-16T04:01:19.845422+0000","last_became_peered":"2024-09-16T04:01:19.845422+0000","last_unstale":"2024-09-16T04:01:55.901578+0000","last_undegraded":"2024-09-16T04:01:55.901578+0000","last_fullsized":"2024-09-16T04:01:55.901578+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:31.239242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036964929000000001,"stat_sum":{"num_bytes":8847360,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":26,"num_read_kb":6985,"num_write":46,"num_write_kb":9168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.e","version":"43'6","reported_seq":186,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.923474+0000","last_change":"2024-09-16T04:01:54.908308+0000","last_active":"2024-09-16T04:01:57.923474+0000","last_peered":"2024-09-16T04:01:57.923474+0000","last_clean":"2024-09-16T04:01:57.923474+0000","last_became_active":"2024-09-16T04:01:20.266249+0000","last_became_peered":"2024-09-16T04:01:20.266249+0000","last_unstale":"2024-09-16T04:01:57.923474+0000","last_undegraded":"2024-09-16T04:01:57.923474+0000","last_fullsized":"2024-09-16T04:01:57.923474+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:42.438971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042994230000000001,"stat_sum":{"num_bytes":1064960,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3317,"num_write":6,"num_write_kb":1440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.d","version":"69'40","reported_seq":219,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.953810+0000","last_change":"2024-09-16T04:01:54.912961+0000","last_active":"2024-09-16T04:01:56.953810+0000","last_peered":"2024-09-16T04:01:56.953810+0000","last_clean":"2024-09-16T04:01:56.953810+0000","last_became_active":"2024-09-16T04:01:19.845009+0000","last_became_peered":"2024-09-16T04:01:19.845009+0000","last_unstale":"2024-09-16T04:01:56.953810+0000","last_undegraded":"2024-09-16T04:01:56.953810+0000","last_fullsized":"2024-09-16T04:01:56.953810+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.998546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023860605,"stat_sum":{"num_bytes":9322496,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":15587,"num_write":40,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.c","version":"70'152","reported_seq":322,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.914582+0000","last_change":"2024-09-16T04:01:54.937443+0000","last_active":"2024-09-16T04:01:55.914582+0000","last_peered":"2024-09-16T04:01:55.914582+0000","last_clean":"2024-09-16T04:01:55.914582+0000","last_became_active":"2024-09-16T04:01:20.259789+0000","last_became_peered":"2024-09-16T04:01:20.259789+0000","last_unstale":"2024-09-16T04:01:55.914582+0000","last_undegraded":"2024-09-16T04:01:55.914582+0000","last_fullsized":"2024-09-16T04:01:55.914582+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:39.754282+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.041360013000000001,"stat_sum":{"num_bytes":17055744,"num_objects":18,"num_object_clones":11,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":48,"num_read_kb":16909,"num_write":134,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.b","version":"69'82","reported_seq":255,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927364+0000","last_change":"2024-09-16T04:01:54.929640+0000","last_active":"2024-09-16T04:01:55.927364+0000","last_peered":"2024-09-16T04:01:55.927364+0000","last_clean":"2024-09-16T04:01:55.927364+0000","last_became_active":"2024-09-16T04:01:20.268548+0000","last_became_peered":"2024-09-16T04:01:20.268548+0000","last_unstale":"2024-09-16T04:01:55.927364+0000","last_undegraded":"2024-09-16T04:01:55.927364+0000","last_fullsized":"2024-09-16T04:01:55.927364+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:21.113712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033503993000000003,"stat_sum":{"num_bytes":12779520,"num_objects":11,"num_object_clones":5,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":40,"num_read_kb":5152,"num_write":64,"num_write_kb":13392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.a","version":"42'10","reported_seq":182,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.274216+0000","last_change":"2024-09-16T04:01:54.891010+0000","last_active":"2024-09-16T04:01:57.274216+0000","last_peered":"2024-09-16T04:01:57.274216+0000","last_clean":"2024-09-16T04:01:57.274216+0000","last_became_active":"2024-09-16T04:01:20.268001+0000","last_became_peered":"2024-09-16T04:01:20.268001+0000","last_unstale":"2024-09-16T04:01:57.274216+0000","last_undegraded":"2024-09-16T04:01:57.274216+0000","last_fullsized":"2024-09-16T04:01:57.274216+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.375530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022714649999999999,"stat_sum":{"num_bytes":704512,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":6196,"num_write":4,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.9","version":"69'21","reported_seq":197,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.943328+0000","last_change":"2024-09-16T04:01:54.917485+0000","last_active":"2024-09-16T04:01:57.943328+0000","last_peered":"2024-09-16T04:01:57.943328+0000","last_clean":"2024-09-16T04:01:57.943328+0000","last_became_active":"2024-09-16T04:01:20.153728+0000","last_became_peered":"2024-09-16T04:01:20.153728+0000","last_unstale":"2024-09-16T04:01:57.943328+0000","last_undegraded":"2024-09-16T04:01:57.943328+0000","last_fullsized":"2024-09-16T04:01:57.943328+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.150743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018434576000000001,"stat_sum":{"num_bytes":2441216,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7157,"num_write":10,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.8","version":"69'15","reported_seq":174,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901158+0000","last_change":"2024-09-16T04:01:54.928232+0000","last_active":"2024-09-16T04:01:55.901158+0000","last_peered":"2024-09-16T04:01:55.901158+0000","last_clean":"2024-09-16T04:01:55.901158+0000","last_became_active":"2024-09-16T04:01:20.259482+0000","last_became_peered":"2024-09-16T04:01:20.259482+0000","last_unstale":"2024-09-16T04:01:55.901158+0000","last_undegraded":"2024-09-16T04:01:55.901158+0000","last_fullsized":"2024-09-16T04:01:55.901158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.552136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017567564000000001,"stat_sum":{"num_bytes":3063808,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":298,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931845+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:56.931845+0000","last_peered":"2024-09-16T04:01:56.931845+0000","last_clean":"2024-09-16T04:01:56.931845+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:56.931845+0000","last_undegraded":"2024-09-16T04:01:56.931845+0000","last_fullsized":"2024-09-16T04:01:56.931845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":96,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901414+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:55.901414+0000","last_peered":"2024-09-16T04:01:55.901414+0000","last_clean":"2024-09-16T04:01:55.901414+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:55.901414+0000","last_undegraded":"2024-09-16T04:01:55.901414+0000","last_fullsized":"2024-09-16T04:01:55.901414+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"65'27","reported_seq":196,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.932186+0000","last_change":"2024-09-16T04:01:54.895830+0000","last_active":"2024-09-16T04:01:56.932186+0000","last_peered":"2024-09-16T04:01:56.932186+0000","last_clean":"2024-09-16T04:01:56.932186+0000","last_became_active":"2024-09-16T04:01:20.258661+0000","last_became_peered":"2024-09-16T04:01:20.258661+0000","last_unstale":"2024-09-16T04:01:56.932186+0000","last_undegraded":"2024-09-16T04:01:56.932186+0000","last_fullsized":"2024-09-16T04:01:56.932186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.280788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001177881,"stat_sum":{"num_bytes":3997696,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":6870,"num_write":18,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":99,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.273943+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:57.273943+0000","last_peered":"2024-09-16T04:01:57.273943+0000","last_clean":"2024-09-16T04:01:57.273943+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:57.273943+0000","last_undegraded":"2024-09-16T04:01:57.273943+0000","last_fullsized":"2024-09-16T04:01:57.273943+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'23","reported_seq":183,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927133+0000","last_change":"2024-09-16T04:01:54.912066+0000","last_active":"2024-09-16T04:01:55.927133+0000","last_peered":"2024-09-16T04:01:55.927133+0000","last_clean":"2024-09-16T04:01:55.927133+0000","last_became_active":"2024-09-16T04:01:20.269891+0000","last_became_peered":"2024-09-16T04:01:20.269891+0000","last_unstale":"2024-09-16T04:01:55.927133+0000","last_undegraded":"2024-09-16T04:01:55.927133+0000","last_fullsized":"2024-09-16T04:01:55.927133+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:33.263117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016000962000000001,"stat_sum":{"num_bytes":2621440,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":1540,"num_write":48,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":101,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931930+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:56.931930+0000","last_peered":"2024-09-16T04:01:56.931930+0000","last_clean":"2024-09-16T04:01:56.931930+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:56.931930+0000","last_undegraded":"2024-09-16T04:01:56.931930+0000","last_fullsized":"2024-09-16T04:01:56.931930+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"69'77","reported_seq":252,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.924464+0000","last_change":"2024-09-16T04:01:54.923402+0000","last_active":"2024-09-16T04:01:57.924464+0000","last_peered":"2024-09-16T04:01:57.924464+0000","last_clean":"2024-09-16T04:01:57.924464+0000","last_became_active":"2024-09-16T04:01:20.257526+0000","last_became_peered":"2024-09-16T04:01:20.257526+0000","last_unstale":"2024-09-16T04:01:57.924464+0000","last_undegraded":"2024-09-16T04:01:57.924464+0000","last_fullsized":"2024-09-16T04:01:57.924464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:21.464739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020041397999999998,"stat_sum":{"num_bytes":8192000,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":9951,"num_write":80,"num_write_kb":11696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":101,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.274219+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:57.274219+0000","last_peered":"2024-09-16T04:01:57.274219+0000","last_clean":"2024-09-16T04:01:57.274219+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:57.274219+0000","last_undegraded":"2024-09-16T04:01:57.274219+0000","last_fullsized":"2024-09-16T04:01:57.274219+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"70'46","reported_seq":216,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.932466+0000","last_change":"2024-09-16T04:01:54.924478+0000","last_active":"2024-09-16T04:01:56.932466+0000","last_peered":"2024-09-16T04:01:56.932466+0000","last_clean":"2024-09-16T04:01:56.932466+0000","last_became_active":"2024-09-16T04:01:19.840099+0000","last_became_peered":"2024-09-16T04:01:19.840099+0000","last_unstale":"2024-09-16T04:01:56.932466+0000","last_undegraded":"2024-09-16T04:01:56.932466+0000","last_fullsized":"2024-09-16T04:01:56.932466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:42.087321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029690392999999999,"stat_sum":{"num_bytes":7700480,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":18,"num_read_kb":12148,"num_write":60,"num_write_kb":7520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.5","version":"70'37","reported_seq":197,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.941837+0000","last_change":"2024-09-16T04:01:54.909344+0000","last_active":"2024-09-16T04:01:55.941837+0000","last_peered":"2024-09-16T04:01:55.941837+0000","last_clean":"2024-09-16T04:01:55.941837+0000","last_became_active":"2024-09-16T04:01:20.258954+0000","last_became_peered":"2024-09-16T04:01:20.258954+0000","last_unstale":"2024-09-16T04:01:55.941837+0000","last_undegraded":"2024-09-16T04:01:55.941837+0000","last_fullsized":"2024-09-16T04:01:55.941837+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.023122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019615982000000001,"stat_sum":{"num_bytes":7225344,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":18,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927159+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:55.927159+0000","last_peered":"2024-09-16T04:01:55.927159+0000","last_clean":"2024-09-16T04:01:55.927159+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:55.927159+0000","last_undegraded":"2024-09-16T04:01:55.927159+0000","last_fullsized":"2024-09-16T04:01:55.927159+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"69'42","reported_seq":219,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.906968+0000","last_change":"2024-09-16T04:01:54.923291+0000","last_active":"2024-09-16T04:01:55.906968+0000","last_peered":"2024-09-16T04:01:55.906968+0000","last_clean":"2024-09-16T04:01:55.906968+0000","last_became_active":"2024-09-16T04:01:20.154175+0000","last_became_peered":"2024-09-16T04:01:20.154175+0000","last_unstale":"2024-09-16T04:01:55.906968+0000","last_undegraded":"2024-09-16T04:01:55.906968+0000","last_fullsized":"2024-09-16T04:01:55.906968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:22.027112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025583108,"stat_sum":{"num_bytes":6062080,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":30,"num_read_kb":5723,"num_write":24,"num_write_kb":7280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:58.000142+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:58.000142+0000","last_peered":"2024-09-16T04:01:58.000142+0000","last_clean":"2024-09-16T04:01:58.000142+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:58.000142+0000","last_undegraded":"2024-09-16T04:01:58.000142+0000","last_fullsized":"2024-09-16T04:01:58.000142+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"36'7","reported_seq":171,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927164+0000","last_change":"2024-09-16T04:01:54.896709+0000","last_active":"2024-09-16T04:01:55.927164+0000","last_peered":"2024-09-16T04:01:55.927164+0000","last_clean":"2024-09-16T04:01:55.927164+0000","last_became_active":"2024-09-16T04:01:20.266522+0000","last_became_peered":"2024-09-16T04:01:20.266522+0000","last_unstale":"2024-09-16T04:01:55.927164+0000","last_undegraded":"2024-09-16T04:01:55.927164+0000","last_fullsized":"2024-09-16T04:01:55.927164+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.553251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000589103,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":724,"num_write":7,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":97,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.906581+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:55.906581+0000","last_peered":"2024-09-16T04:01:55.906581+0000","last_clean":"2024-09-16T04:01:55.906581+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:55.906581+0000","last_undegraded":"2024-09-16T04:01:55.906581+0000","last_fullsized":"2024-09-16T04:01:55.906581+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"36'2","reported_seq":168,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931956+0000","last_change":"2024-09-16T04:01:54.894048+0000","last_active":"2024-09-16T04:01:56.931956+0000","last_peered":"2024-09-16T04:01:56.931956+0000","last_clean":"2024-09-16T04:01:56.931956+0000","last_became_active":"2024-09-16T04:01:20.267489+0000","last_became_peered":"2024-09-16T04:01:20.267489+0000","last_unstale":"2024-09-16T04:01:56.931956+0000","last_undegraded":"2024-09-16T04:01:56.931956+0000","last_fullsized":"2024-09-16T04:01:56.931956+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:41.900963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078888799999999996,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2499,"num_write":2,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.897011+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:55.897011+0000","last_peered":"2024-09-16T04:01:55.897011+0000","last_clean":"2024-09-16T04:01:55.897011+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:55.897011+0000","last_undegraded":"2024-09-16T04:01:55.897011+0000","last_fullsized":"2024-09-16T04:01:55.897011+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92782592,"num_objects":91,"num_object_clones":42,"num_object_copies":637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":3,"num_read":320,"num_read_kb":100766,"num_write":570,"num_write_kb":100416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167907328,"data_stored":233668608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":634,"ondisk_log_size":634,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42040,"kb_used_data":10848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329800,"statfs":{"total":96636764160,"available":96593715200,"internally_reserved":0,"allocated":11108352,"data_stored":12998221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215120,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42444,"kb_used_data":11252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329396,"statfs":{"total":96636764160,"available":96593301504,"internally_reserved":0,"allocated":11522048,"data_stored":16832077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215121,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45676,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326164,"statfs":{"total":96636764160,"available":96589991936,"internally_reserved":0,"allocated":14831616,"data_stored":20842061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37424,"kb_used_data":10392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334416,"statfs":{"total":96636764160,"available":96598441984,"internally_reserved":0,"allocated":10641408,"data_stored":12002743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280529,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45240,"kb_used_data":14048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326600,"statfs":{"total":96636764160,"available":96590438400,"internally_reserved":0,"allocated":14385152,"data_stored":17157584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280527,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40888,"kb_used_data":13856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330952,"statfs":{"total":96636764160,"available":96594894848,"internally_reserved":0,"allocated":14188544,"data_stored":19269047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280527,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":5544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":5677056,"data_stored":5698999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280527,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":9636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":9867264,"data_stored":14097872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280527,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35620,"kb_used_data":8588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336220,"statfs":{"total":96636764160,"available":96600289280,"internally_reserved":0,"allocated":8794112,"data_stored":12793290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35500,"kb_used_data":8468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336340,"statfs":{"total":96636764160,"available":96600412160,"internally_reserved":0,"allocated":8671232,"data_stored":10634679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215120,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43284,"kb_used_data":12096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328556,"statfs":{"total":96636764160,"available":96592441344,"internally_reserved":0,"allocated":12386304,"data_stored":14766214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48864,"kb_used_data":17676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322976,"statfs":{"total":96636764160,"available":96586727424,"internally_reserved":0,"allocated":18100224,"data_stored":24260048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247824,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46292,"kb_used_data":10940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325548,"statfs":{"total":96636764160,"available":96589361152,"internally_reserved":0,"allocated":11202560,"data_stored":13232237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247824,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38044,"kb_used_data":6856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333796,"statfs":{"total":96636764160,"available":96597807104,"internally_reserved":0,"allocated":7020544,"data_stored":8314339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41884,"kb_used_data":10696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329956,"statfs":{"total":96636764160,"available":96593874944,"internally_reserved":0,"allocated":10952704,"data_stored":17194448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41456,"kb_used_data":10268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330384,"statfs":{"total":96636764160,"available":96594313216,"internally_reserved":0,"allocated":10514432,"data_stored":12332496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17584128,"data_stored":24752128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9682944,"data_stored":11468800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10846208,"data_stored":14147584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6041600,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13795328,"data_stored":16998400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9977856,"data_stored":16650240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14991360,"data_stored":22626304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9539584,"data_stored":11960320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13324288,"data_stored":19214336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4718592,"data_stored":5165056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":12152832,"data_stored":19853312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8892416,"data_stored":13553664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":7831552,"data_stored":12644352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":9678848,"data_stored":12148736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10588160,"data_stored":13680640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":8261632,"data_stored":11034624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:01:59.696 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T04:01:59.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:01:59.696 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-16T04:01:59.697 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:01:59.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:01:59.783 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:01:59.781+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:01:59.822 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:59.823 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:59.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:01:59.884 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:01:59.877+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:01:59.938 INFO:tasks.rados.rados.0.smithi188.stdout:266: done (1 left) 2024-09-16T04:01:59.938 INFO:tasks.rados.rados.0.smithi188.stdout:267: done (0 left) 2024-09-16T04:01:59.938 INFO:tasks.rados.rados.0.smithi188.stdout:268: read oid 17 snap -1 2024-09-16T04:01:59.938 INFO:tasks.rados.rados.0.smithi188.stdout:268: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:01:59.938 INFO:tasks.rados.rados.0.smithi188.stdout:269: read oid 24 snap 12 2024-09-16T04:01:59.938 INFO:tasks.rados.rados.0.smithi188.stdout:269: expect (ObjNum 70 snap 3 seq_num 70) 2024-09-16T04:01:59.938 INFO:tasks.rados.rados.0.smithi188.stdout:270: snap_create 2024-09-16T04:01:59.956 INFO:tasks.rados.rados.0.smithi188.stdout:269: expect (ObjNum 70 snap 3 seq_num 70) 2024-09-16T04:01:59.958 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:01:59.958 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:01:59.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:01:59.985 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:01:59.981+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:02:00.043 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:02:00.044 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:02:00.054 INFO:tasks.rados.rados.0.smithi188.stdout:268: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:02:00.061 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-16T04:01:59.934094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93372979,"num_objects":95,"num_object_clones":42,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":3,"num_read":456,"num_read_kb":100884,"num_write":775,"num_write_kb":103878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":653904,"kb_used_data":175648,"kb_used_omap":24,"kb_used_meta":477863,"kb_avail":1509295536,"statfs":{"total":1546188226560,"available":1545518628864,"internally_reserved":0,"allocated":179863552,"data_stored":232426355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":489331793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15597568,"num_objects":9,"num_object_clones":9,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":-2,"num_read":34,"num_read_kb":20271,"num_write":34,"num_write_kb":15232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.033425"},"pg_stats":[{"pgid":"3.f","version":"69'47","reported_seq":222,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901578+0000","last_change":"2024-09-16T04:01:54.932899+0000","last_active":"2024-09-16T04:01:55.901578+0000","last_peered":"2024-09-16T04:01:55.901578+0000","last_clean":"2024-09-16T04:01:55.901578+0000","last_became_active":"2024-09-16T04:01:19.845422+0000","last_became_peered":"2024-09-16T04:01:19.845422+0000","last_unstale":"2024-09-16T04:01:55.901578+0000","last_undegraded":"2024-09-16T04:01:55.901578+0000","last_fullsized":"2024-09-16T04:01:55.901578+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:31.239242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036964929000000001,"stat_sum":{"num_bytes":8847360,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":26,"num_read_kb":6985,"num_write":46,"num_write_kb":9168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.e","version":"43'6","reported_seq":186,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.923474+0000","last_change":"2024-09-16T04:01:54.908308+0000","last_active":"2024-09-16T04:01:57.923474+0000","last_peered":"2024-09-16T04:01:57.923474+0000","last_clean":"2024-09-16T04:01:57.923474+0000","last_became_active":"2024-09-16T04:01:20.266249+0000","last_became_peered":"2024-09-16T04:01:20.266249+0000","last_unstale":"2024-09-16T04:01:57.923474+0000","last_undegraded":"2024-09-16T04:01:57.923474+0000","last_fullsized":"2024-09-16T04:01:57.923474+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:42.438971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042994230000000001,"stat_sum":{"num_bytes":1064960,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3317,"num_write":6,"num_write_kb":1440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.d","version":"69'40","reported_seq":219,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.953810+0000","last_change":"2024-09-16T04:01:54.912961+0000","last_active":"2024-09-16T04:01:56.953810+0000","last_peered":"2024-09-16T04:01:56.953810+0000","last_clean":"2024-09-16T04:01:56.953810+0000","last_became_active":"2024-09-16T04:01:19.845009+0000","last_became_peered":"2024-09-16T04:01:19.845009+0000","last_unstale":"2024-09-16T04:01:56.953810+0000","last_undegraded":"2024-09-16T04:01:56.953810+0000","last_fullsized":"2024-09-16T04:01:56.953810+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.998546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023860605,"stat_sum":{"num_bytes":9322496,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":15587,"num_write":40,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.c","version":"70'152","reported_seq":322,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.914582+0000","last_change":"2024-09-16T04:01:54.937443+0000","last_active":"2024-09-16T04:01:55.914582+0000","last_peered":"2024-09-16T04:01:55.914582+0000","last_clean":"2024-09-16T04:01:55.914582+0000","last_became_active":"2024-09-16T04:01:20.259789+0000","last_became_peered":"2024-09-16T04:01:20.259789+0000","last_unstale":"2024-09-16T04:01:55.914582+0000","last_undegraded":"2024-09-16T04:01:55.914582+0000","last_fullsized":"2024-09-16T04:01:55.914582+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:39.754282+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.041360013000000001,"stat_sum":{"num_bytes":17055744,"num_objects":18,"num_object_clones":11,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":48,"num_read_kb":16909,"num_write":134,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.b","version":"69'82","reported_seq":255,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927364+0000","last_change":"2024-09-16T04:01:54.929640+0000","last_active":"2024-09-16T04:01:55.927364+0000","last_peered":"2024-09-16T04:01:55.927364+0000","last_clean":"2024-09-16T04:01:55.927364+0000","last_became_active":"2024-09-16T04:01:20.268548+0000","last_became_peered":"2024-09-16T04:01:20.268548+0000","last_unstale":"2024-09-16T04:01:55.927364+0000","last_undegraded":"2024-09-16T04:01:55.927364+0000","last_fullsized":"2024-09-16T04:01:55.927364+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:21.113712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033503993000000003,"stat_sum":{"num_bytes":12779520,"num_objects":11,"num_object_clones":5,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":40,"num_read_kb":5152,"num_write":64,"num_write_kb":13392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.a","version":"42'10","reported_seq":182,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.274216+0000","last_change":"2024-09-16T04:01:54.891010+0000","last_active":"2024-09-16T04:01:57.274216+0000","last_peered":"2024-09-16T04:01:57.274216+0000","last_clean":"2024-09-16T04:01:57.274216+0000","last_became_active":"2024-09-16T04:01:20.268001+0000","last_became_peered":"2024-09-16T04:01:20.268001+0000","last_unstale":"2024-09-16T04:01:57.274216+0000","last_undegraded":"2024-09-16T04:01:57.274216+0000","last_fullsized":"2024-09-16T04:01:57.274216+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.375530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022714649999999999,"stat_sum":{"num_bytes":704512,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":6196,"num_write":4,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.9","version":"69'21","reported_seq":197,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.943328+0000","last_change":"2024-09-16T04:01:54.917485+0000","last_active":"2024-09-16T04:01:57.943328+0000","last_peered":"2024-09-16T04:01:57.943328+0000","last_clean":"2024-09-16T04:01:57.943328+0000","last_became_active":"2024-09-16T04:01:20.153728+0000","last_became_peered":"2024-09-16T04:01:20.153728+0000","last_unstale":"2024-09-16T04:01:57.943328+0000","last_undegraded":"2024-09-16T04:01:57.943328+0000","last_fullsized":"2024-09-16T04:01:57.943328+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.150743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018434576000000001,"stat_sum":{"num_bytes":2441216,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7157,"num_write":10,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.8","version":"69'15","reported_seq":174,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901158+0000","last_change":"2024-09-16T04:01:54.928232+0000","last_active":"2024-09-16T04:01:55.901158+0000","last_peered":"2024-09-16T04:01:55.901158+0000","last_clean":"2024-09-16T04:01:55.901158+0000","last_became_active":"2024-09-16T04:01:20.259482+0000","last_became_peered":"2024-09-16T04:01:20.259482+0000","last_unstale":"2024-09-16T04:01:55.901158+0000","last_undegraded":"2024-09-16T04:01:55.901158+0000","last_fullsized":"2024-09-16T04:01:55.901158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.552136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017567564000000001,"stat_sum":{"num_bytes":3063808,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":298,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931845+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:01:56.931845+0000","last_peered":"2024-09-16T04:01:56.931845+0000","last_clean":"2024-09-16T04:01:56.931845+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:01:56.931845+0000","last_undegraded":"2024-09-16T04:01:56.931845+0000","last_fullsized":"2024-09-16T04:01:56.931845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":96,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.901414+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:01:55.901414+0000","last_peered":"2024-09-16T04:01:55.901414+0000","last_clean":"2024-09-16T04:01:55.901414+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:01:55.901414+0000","last_undegraded":"2024-09-16T04:01:55.901414+0000","last_fullsized":"2024-09-16T04:01:55.901414+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"65'27","reported_seq":196,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.932186+0000","last_change":"2024-09-16T04:01:54.895830+0000","last_active":"2024-09-16T04:01:56.932186+0000","last_peered":"2024-09-16T04:01:56.932186+0000","last_clean":"2024-09-16T04:01:56.932186+0000","last_became_active":"2024-09-16T04:01:20.258661+0000","last_became_peered":"2024-09-16T04:01:20.258661+0000","last_unstale":"2024-09-16T04:01:56.932186+0000","last_undegraded":"2024-09-16T04:01:56.932186+0000","last_fullsized":"2024-09-16T04:01:56.932186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.280788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001177881,"stat_sum":{"num_bytes":3997696,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":6870,"num_write":18,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":99,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.273943+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:01:57.273943+0000","last_peered":"2024-09-16T04:01:57.273943+0000","last_clean":"2024-09-16T04:01:57.273943+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:01:57.273943+0000","last_undegraded":"2024-09-16T04:01:57.273943+0000","last_fullsized":"2024-09-16T04:01:57.273943+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'23","reported_seq":183,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927133+0000","last_change":"2024-09-16T04:01:54.912066+0000","last_active":"2024-09-16T04:01:55.927133+0000","last_peered":"2024-09-16T04:01:55.927133+0000","last_clean":"2024-09-16T04:01:55.927133+0000","last_became_active":"2024-09-16T04:01:20.269891+0000","last_became_peered":"2024-09-16T04:01:20.269891+0000","last_unstale":"2024-09-16T04:01:55.927133+0000","last_undegraded":"2024-09-16T04:01:55.927133+0000","last_fullsized":"2024-09-16T04:01:55.927133+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:33.263117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016000962000000001,"stat_sum":{"num_bytes":2621440,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":1540,"num_write":48,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":101,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931930+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:01:56.931930+0000","last_peered":"2024-09-16T04:01:56.931930+0000","last_clean":"2024-09-16T04:01:56.931930+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:01:56.931930+0000","last_undegraded":"2024-09-16T04:01:56.931930+0000","last_fullsized":"2024-09-16T04:01:56.931930+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"69'77","reported_seq":252,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.924464+0000","last_change":"2024-09-16T04:01:54.923402+0000","last_active":"2024-09-16T04:01:57.924464+0000","last_peered":"2024-09-16T04:01:57.924464+0000","last_clean":"2024-09-16T04:01:57.924464+0000","last_became_active":"2024-09-16T04:01:20.257526+0000","last_became_peered":"2024-09-16T04:01:20.257526+0000","last_unstale":"2024-09-16T04:01:57.924464+0000","last_undegraded":"2024-09-16T04:01:57.924464+0000","last_fullsized":"2024-09-16T04:01:57.924464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:21.464739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020041397999999998,"stat_sum":{"num_bytes":8192000,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":9951,"num_write":80,"num_write_kb":11696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":101,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:57.274219+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:01:57.274219+0000","last_peered":"2024-09-16T04:01:57.274219+0000","last_clean":"2024-09-16T04:01:57.274219+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:01:57.274219+0000","last_undegraded":"2024-09-16T04:01:57.274219+0000","last_fullsized":"2024-09-16T04:01:57.274219+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"70'46","reported_seq":216,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.932466+0000","last_change":"2024-09-16T04:01:54.924478+0000","last_active":"2024-09-16T04:01:56.932466+0000","last_peered":"2024-09-16T04:01:56.932466+0000","last_clean":"2024-09-16T04:01:56.932466+0000","last_became_active":"2024-09-16T04:01:19.840099+0000","last_became_peered":"2024-09-16T04:01:19.840099+0000","last_unstale":"2024-09-16T04:01:56.932466+0000","last_undegraded":"2024-09-16T04:01:56.932466+0000","last_fullsized":"2024-09-16T04:01:56.932466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:42.087321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029690392999999999,"stat_sum":{"num_bytes":7700480,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":18,"num_read_kb":12148,"num_write":60,"num_write_kb":7520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"3.5","version":"70'37","reported_seq":197,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.941837+0000","last_change":"2024-09-16T04:01:54.909344+0000","last_active":"2024-09-16T04:01:55.941837+0000","last_peered":"2024-09-16T04:01:55.941837+0000","last_clean":"2024-09-16T04:01:55.941837+0000","last_became_active":"2024-09-16T04:01:20.258954+0000","last_became_peered":"2024-09-16T04:01:20.258954+0000","last_unstale":"2024-09-16T04:01:55.941837+0000","last_undegraded":"2024-09-16T04:01:55.941837+0000","last_fullsized":"2024-09-16T04:01:55.941837+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.023122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019615982000000001,"stat_sum":{"num_bytes":7225344,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":18,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927159+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:01:55.927159+0000","last_peered":"2024-09-16T04:01:55.927159+0000","last_clean":"2024-09-16T04:01:55.927159+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:01:55.927159+0000","last_undegraded":"2024-09-16T04:01:55.927159+0000","last_fullsized":"2024-09-16T04:01:55.927159+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"69'42","reported_seq":219,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.906968+0000","last_change":"2024-09-16T04:01:54.923291+0000","last_active":"2024-09-16T04:01:55.906968+0000","last_peered":"2024-09-16T04:01:55.906968+0000","last_clean":"2024-09-16T04:01:55.906968+0000","last_became_active":"2024-09-16T04:01:20.154175+0000","last_became_peered":"2024-09-16T04:01:20.154175+0000","last_unstale":"2024-09-16T04:01:55.906968+0000","last_undegraded":"2024-09-16T04:01:55.906968+0000","last_fullsized":"2024-09-16T04:01:55.906968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:22.027112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025583108,"stat_sum":{"num_bytes":6062080,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":30,"num_read_kb":5723,"num_write":24,"num_write_kb":7280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T04:01:58.000142+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:01:58.000142+0000","last_peered":"2024-09-16T04:01:58.000142+0000","last_clean":"2024-09-16T04:01:58.000142+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:01:58.000142+0000","last_undegraded":"2024-09-16T04:01:58.000142+0000","last_fullsized":"2024-09-16T04:01:58.000142+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"36'7","reported_seq":171,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.927164+0000","last_change":"2024-09-16T04:01:54.896709+0000","last_active":"2024-09-16T04:01:55.927164+0000","last_peered":"2024-09-16T04:01:55.927164+0000","last_clean":"2024-09-16T04:01:55.927164+0000","last_became_active":"2024-09-16T04:01:20.266522+0000","last_became_peered":"2024-09-16T04:01:20.266522+0000","last_unstale":"2024-09-16T04:01:55.927164+0000","last_undegraded":"2024-09-16T04:01:55.927164+0000","last_fullsized":"2024-09-16T04:01:55.927164+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.553251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000589103,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":724,"num_write":7,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":97,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.906581+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:01:55.906581+0000","last_peered":"2024-09-16T04:01:55.906581+0000","last_clean":"2024-09-16T04:01:55.906581+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:01:55.906581+0000","last_undegraded":"2024-09-16T04:01:55.906581+0000","last_fullsized":"2024-09-16T04:01:55.906581+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"36'2","reported_seq":168,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T04:01:56.931956+0000","last_change":"2024-09-16T04:01:54.894048+0000","last_active":"2024-09-16T04:01:56.931956+0000","last_peered":"2024-09-16T04:01:56.931956+0000","last_clean":"2024-09-16T04:01:56.931956+0000","last_became_active":"2024-09-16T04:01:20.267489+0000","last_became_peered":"2024-09-16T04:01:20.267489+0000","last_unstale":"2024-09-16T04:01:56.931956+0000","last_undegraded":"2024-09-16T04:01:56.931956+0000","last_fullsized":"2024-09-16T04:01:56.931956+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:41.900963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078888799999999996,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2499,"num_write":2,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T04:01:55.897011+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:01:55.897011+0000","last_peered":"2024-09-16T04:01:55.897011+0000","last_clean":"2024-09-16T04:01:55.897011+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:01:55.897011+0000","last_undegraded":"2024-09-16T04:01:55.897011+0000","last_fullsized":"2024-09-16T04:01:55.897011+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92782592,"num_objects":91,"num_object_clones":42,"num_object_copies":637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":3,"num_read":320,"num_read_kb":100766,"num_write":570,"num_write_kb":100416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167907328,"data_stored":233668608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":634,"ondisk_log_size":634,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42040,"kb_used_data":10848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329800,"statfs":{"total":96636764160,"available":96593715200,"internally_reserved":0,"allocated":11108352,"data_stored":12998221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215120,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42444,"kb_used_data":11252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329396,"statfs":{"total":96636764160,"available":96593301504,"internally_reserved":0,"allocated":11522048,"data_stored":16832077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215121,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45676,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326164,"statfs":{"total":96636764160,"available":96589991936,"internally_reserved":0,"allocated":14831616,"data_stored":20842061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37424,"kb_used_data":10392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334416,"statfs":{"total":96636764160,"available":96598441984,"internally_reserved":0,"allocated":10641408,"data_stored":12002743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280529,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45240,"kb_used_data":14048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326600,"statfs":{"total":96636764160,"available":96590438400,"internally_reserved":0,"allocated":14385152,"data_stored":17157584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280527,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40888,"kb_used_data":13856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330952,"statfs":{"total":96636764160,"available":96594894848,"internally_reserved":0,"allocated":14188544,"data_stored":19269047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280527,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":5544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":5677056,"data_stored":5698999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280527,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":9636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":9867264,"data_stored":14097872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280527,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35620,"kb_used_data":8588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336220,"statfs":{"total":96636764160,"available":96600289280,"internally_reserved":0,"allocated":8794112,"data_stored":12793290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35500,"kb_used_data":8468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336340,"statfs":{"total":96636764160,"available":96600412160,"internally_reserved":0,"allocated":8671232,"data_stored":10634679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215120,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43284,"kb_used_data":12096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328556,"statfs":{"total":96636764160,"available":96592441344,"internally_reserved":0,"allocated":12386304,"data_stored":14766214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48864,"kb_used_data":17676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322976,"statfs":{"total":96636764160,"available":96586727424,"internally_reserved":0,"allocated":18100224,"data_stored":24260048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247824,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46292,"kb_used_data":10940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325548,"statfs":{"total":96636764160,"available":96589361152,"internally_reserved":0,"allocated":11202560,"data_stored":13232237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247824,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38044,"kb_used_data":6856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333796,"statfs":{"total":96636764160,"available":96597807104,"internally_reserved":0,"allocated":7020544,"data_stored":8314339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41884,"kb_used_data":10696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329956,"statfs":{"total":96636764160,"available":96593874944,"internally_reserved":0,"allocated":10952704,"data_stored":17194448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41456,"kb_used_data":10268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330384,"statfs":{"total":96636764160,"available":96594313216,"internally_reserved":0,"allocated":10514432,"data_stored":12332496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17584128,"data_stored":24752128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9682944,"data_stored":11468800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10846208,"data_stored":14147584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6041600,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13795328,"data_stored":16998400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9977856,"data_stored":16650240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14991360,"data_stored":22626304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9539584,"data_stored":11960320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13324288,"data_stored":19214336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4718592,"data_stored":5165056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":12152832,"data_stored":19853312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8892416,"data_stored":13553664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":7831552,"data_stored":12644352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":9678848,"data_stored":12148736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10588160,"data_stored":13680640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":8261632,"data_stored":11034624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:02:00.063 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-09-16T04:02:00.086 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:00.081+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:00.094 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:00.094 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:00.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:02:00.187 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:00.181+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:00.237 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:00.237 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:00.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:02:00.296 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:00.293+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:00.339 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:02:00.373 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:00.373 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:00.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:02:00.389 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:00.385+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:00.486 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:00.487 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:00.489 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:00.484+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:00.507 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:00.527 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:00.527 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:00.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:00.557 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:00.557 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:00.565 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:00.565 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:00.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:02:00.585 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:00.586 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:00.590 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:00.589+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:00.605 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:00.605 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:00.626 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:00.626 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:00.667 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:00.667 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:00.690 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:00.690 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:00.691 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:00.689+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 39/33 ECSubWrite(tid=80, reqid=client.4646.0:291, at_version=39'41, trim_to=0'0, roll_forward_to=38'39))", 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 35.800848733999999, 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062540319999999996, 2024-09-16T04:02:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.888599+0000", 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.13e-07 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889239+0000", 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000639432 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889269+0000", 2024-09-16T04:02:00.695 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0131999999999999e-05 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889293+0000", 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4396000000000002e-05 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889486+0000", 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019278000000000001 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.696 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.889535+0000", 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8442000000000001e-05 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894775+0000", 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052406800000000002 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:24.894853+0000", 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7557000000000003e-05 2024-09-16T04:02:00.697 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/33 ECSubWrite(tid=70, reqid=osd.4.0:70, at_version=41'12, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 33.789127299999997, 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059225500000000004, 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.698 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900320+0000", 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900322+0000", 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.894e-06 2024-09-16T04:02:00.699 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900375+0000", 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2994000000000001e-05 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900405+0000", 2024-09-16T04:02:00.700 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9543000000000002e-05 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.900447+0000", 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.2824999999999997e-05 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902403+0000", 2024-09-16T04:02:00.701 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001955666 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902429+0000", 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5783e-05 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906151+0000", 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037224810000000001 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.702 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906243+0000", 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.1364000000000001e-05 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.ds3 41/33 ECSubWrite(tid=29, reqid=osd.7.0:29, at_version=41'19, trim_to=0'0, roll_forward_to=38'15))", 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 33.780812902000001, 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053361119999999996, 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.703 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908635+0000", 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.704 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908637+0000", 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9269999999999999e-06 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908688+0000", 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:02:00.705 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908719+0000", 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1479000000000001e-05 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908741+0000", 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2152999999999999e-05 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908934+0000", 2024-09-16T04:02:00.706 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019235000000000001 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.908957+0000", 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3572000000000001e-05 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.913883+0000", 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0049257820000000001 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.707 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.913971+0000", 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7854000000000003e-05 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/33 ECSubWrite(tid=101, reqid=osd.11.0:101, at_version=41'54, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 33.775430694000001, 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062086830000000004, 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.708 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.709 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914017+0000", 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7999999999999995e-07 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914081+0000", 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.3151000000000006e-05 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914110+0000", 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9672e-05 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914133+0000", 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2439e-05 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914712+0000", 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00057882500000000004 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914737+0000", 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:02:00.711 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920147+0000", 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0054103859999999997 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920226+0000", 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8573999999999997e-05 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.712 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 41/33 ECSubWrite(tid=100, reqid=osd.11.0:100, at_version=41'21, trim_to=0'0, roll_forward_to=40'19))", 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 33.775259929999997, 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0078425089999999992, 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:02:00.713 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914188+0000", 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914189+0000", 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2999999999999999e-07 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.714 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914239+0000", 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0232000000000003e-05 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914261+0000", 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2282000000000001e-05 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.914279+0000", 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8283000000000001e-05 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.715 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915647+0000", 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0013673719999999999 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915669+0000", 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2198999999999999e-05 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921948+0000", 2024-09-16T04:02:00.716 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062792710000000003 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922030+0000", 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1940000000000005e-05 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/33 ECSubWrite(tid=72, reqid=osd.4.0:70, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 33.774398501999997, 2024-09-16T04:02:00.717 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0075338999999999996, 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915049+0000", 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.718 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915051+0000", 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.057e-06 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915081+0000", 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9488999999999998e-05 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915114+0000", 2024-09-16T04:02:00.719 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3312000000000002e-05 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915138+0000", 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4450000000000001e-05 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915351+0000", 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00021263900000000001 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.720 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.915375+0000", 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4219999999999999e-05 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922495+0000", 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0071197559999999997 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.922583+0000", 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7977000000000001e-05 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.721 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 33.772553922, 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0069081630000000002, 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:02:00.722 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916894+0000", 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916895+0000", 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.88e-06 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.723 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916949+0000", 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3551999999999998e-05 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916983+0000", 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4178999999999999e-05 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917008+0000", 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5049e-05 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.724 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917176+0000", 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016821800000000001 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917201+0000", 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4695999999999999e-05 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923709+0000", 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0065079020000000003 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.725 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.923802+0000", 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2687000000000002e-05 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.9s6 51/33 ECSubWrite(tid=75, reqid=client.4646.0:384, at_version=51'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.852058+0000", 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 23.837389434999999, 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0070891699999999997, 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.726 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852058+0000", 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852058+0000", 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.727 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852058+0000", 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3500000000000002e-07 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852200+0000", 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014109499999999999 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.728 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852218+0000", 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8380999999999999e-05 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852229+0000", 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1545999999999999e-05 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852297+0000", 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7811000000000005e-05 2024-09-16T04:02:00.729 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.852304+0000", 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6250000000000001e-06 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.859119+0000", 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006815354 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.859147+0000", 2024-09-16T04:02:00.730 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8022999999999999e-05 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=177, reqid=client.4646.0:449, at_version=60'27, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.745511+0000", 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 14.943936939, 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0067418640000000002, 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745511+0000", 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745511+0000", 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.745511+0000", 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3e-07 2024-09-16T04:02:00.732 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746037+0000", 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00052584300000000001 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746075+0000", 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7595999999999997e-05 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746101+0000", 2024-09-16T04:02:00.733 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6375000000000001e-05 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746354+0000", 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000253212 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.746382+0000", 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7912999999999999e-05 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.752157+0000", 2024-09-16T04:02:00.734 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057748039999999997 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.752252+0000", 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5390999999999995e-05 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=185, reqid=client.4646.0:464, at_version=60'34, trim_to=0'0, roll_forward_to=60'32))", 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.791419+0000", 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 14.898028776, 2024-09-16T04:02:00.735 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0084625380000000004, 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791419+0000", 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791419+0000", 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.736 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791421+0000", 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.17e-06 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791652+0000", 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023153200000000001 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.737 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791701+0000", 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8721999999999999e-05 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.791726+0000", 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4992000000000001e-05 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.792082+0000", 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00035561700000000002 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.738 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.795537+0000", 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0034547710000000001 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.799810+0000", 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042732020000000003 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.799881+0000", 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1532000000000006e-05 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.739 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=187, reqid=client.4646.0:465, at_version=60'35, trim_to=0'0, roll_forward_to=60'32))", 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.796267+0000", 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 14.89318081, 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0081368080000000006, 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.740 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796267+0000", 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796267+0000", 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796269+0000", 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.741 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796395+0000", 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012676699999999999 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796430+0000", 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4303e-05 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796453+0000", 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3603000000000001e-05 2024-09-16T04:02:00.742 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796862+0000", 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000408568 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796887+0000", 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5109e-05 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.743 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.804291+0000", 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0074039270000000003 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.804404+0000", 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000112551 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/33 ECSubWrite(tid=189, reqid=client.4646.0:466, at_version=60'36, trim_to=0'0, roll_forward_to=60'32))", 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:45.796500+0000", 2024-09-16T04:02:00.744 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 14.892947577999999, 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079727640000000002, 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796500+0000", 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.745 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796500+0000", 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796501+0000", 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1000000000000001e-06 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796532+0000", 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0549000000000002e-05 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.746 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796556+0000", 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4046e-05 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.796584+0000", 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7949999999999998e-05 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.798085+0000", 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001501299 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.747 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.798109+0000", 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4326000000000001e-05 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.804414+0000", 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0063049660000000004 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:45.804473+0000", 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8527999999999999e-05 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.748 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 61/33 ECSubWrite(tid=194, reqid=client.4646.0:472, at_version=61'118, trim_to=0'0, roll_forward_to=60'116))", 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:46.744103+0000", 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 13.94534475, 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055045090000000003, 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.749 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744103+0000", 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744103+0000", 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744104+0000", 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7300000000000005e-07 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.750 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.744985+0000", 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00088172800000000003 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745010+0000", 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5179999999999999e-05 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745029+0000", 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8036000000000001e-05 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.751 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745205+0000", 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017667 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.745222+0000", 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6776e-05 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.752 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.749536+0000", 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0043142340000000001 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:46.749607+0000", 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1112000000000006e-05 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/33 ECSubWrite(tid=216, reqid=client.4646.0:540, at_version=65'38, trim_to=0'0, roll_forward_to=60'36))", 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.095329+0000", 2024-09-16T04:02:00.753 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.5941186849999998, 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0068702520000000003, 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.095329+0000", 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.095329+0000", 2024-09-16T04:02:00.754 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.095331+0000", 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8989999999999999e-06 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.097846+0000", 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0025153139999999998 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.097874+0000", 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7617999999999999e-05 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.097890+0000", 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6456e-05 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.756 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.097998+0000", 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000108162 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.098009+0000", 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.041e-05 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.102158+0000", 2024-09-16T04:02:00.757 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004149224 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.102199+0000", 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1168999999999998e-05 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 66/33 ECSubWrite(tid=139, reqid=osd.4.0:139, at_version=66'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.727946+0000", 2024-09-16T04:02:00.758 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 8.9615011659999997, 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0092200370000000004, 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727946+0000", 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.759 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727946+0000", 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727947+0000", 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8400000000000004e-07 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.760 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727980+0000", 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2691000000000003e-05 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.727999+0000", 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8987e-05 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.728012+0000", 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3682e-05 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.761 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.728107+0000", 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.4248999999999998e-05 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.728119+0000", 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2194e-05 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.737120+0000", 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0090007349999999993 2024-09-16T04:02:00.762 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.737166+0000", 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.6814999999999998e-05 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.763 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 66/33 ECSubWrite(tid=142, reqid=osd.4.0:139, at_version=0'0, trim_to=0'0, roll_forward_to=66'21))", 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:51.741702+0000", 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 8.9477455829999997, 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062051329999999998, 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741702+0000", 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.764 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741702+0000", 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741703+0000", 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0139999999999999e-06 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741717+0000", 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3557000000000001e-05 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741734+0000", 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.753e-05 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741745+0000", 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0451999999999999e-05 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.766 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741838+0000", 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.3114000000000003e-05 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.741847+0000", 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.064e-06 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.747864+0000", 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.006016965 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:51.747907+0000", 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3436999999999998e-05 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 70/33 ECSubWrite(tid=248, reqid=client.4646.0:609, at_version=70'150, trim_to=0'0, roll_forward_to=69'148))", 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:55.904217+0000", 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 4.7852302240000002, 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0081934190000000004, 2024-09-16T04:02:00.768 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904217+0000", 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904217+0000", 2024-09-16T04:02:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904218+0000", 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3600000000000003e-07 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904277+0000", 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8505999999999999e-05 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904311+0000", 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4898999999999999e-05 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904344+0000", 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2678999999999997e-05 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904725+0000", 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00038050099999999999 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.771 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904751+0000", 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6210000000000001e-05 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.912343+0000", 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0075916630000000002 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.912411+0000", 2024-09-16T04:02:00.772 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8225000000000005e-05 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 70/33 ECSubWrite(tid=249, reqid=client.4646.0:610, at_version=70'151, trim_to=0'0, roll_forward_to=69'148))", 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:55.907369+0000", 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 4.7820783179999999, 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0060081570000000001, 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.773 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.907369+0000", 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.907369+0000", 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.907371+0000", 2024-09-16T04:02:00.774 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.626e-06 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.907421+0000", 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0544999999999998e-05 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.907454+0000", 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2762999999999998e-05 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.775 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.907479+0000", 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4732999999999999e-05 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.907660+0000", 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000181266 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.907685+0000", 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5072e-05 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.913302+0000", 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0056164370000000002 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.913377+0000", 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.5715000000000003e-05 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.777 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 71/33 ECSubWrite(tid=252, reqid=client.4646.0:624, at_version=71'49, trim_to=0'0, roll_forward_to=69'47))", 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:56.930590+0000", 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.7588575230000001, 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0066152140000000003, 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.930590+0000", 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.778 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.930590+0000", 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.930591+0000", 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.930971+0000", 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00037970799999999998 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.779 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.931005+0000", 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4473e-05 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.931029+0000", 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4145999999999999e-05 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.931199+0000", 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000169836 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.780 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.931224+0000", 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5358999999999999e-05 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.937125+0000", 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059001000000000001 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:56.937205+0000", 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0692000000000006e-05 2024-09-16T04:02:00.781 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 72/33 ECSubWrite(tid=258, reqid=client.4646.0:644, at_version=72'154, trim_to=0'0, roll_forward_to=70'152))", 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.997022+0000", 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 2.6924255490000002, 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0066776309999999998, 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.782 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.997022+0000", 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.997022+0000", 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.997023+0000", 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7700000000000004e-07 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.783 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.997500+0000", 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00047736499999999999 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.997532+0000", 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1625999999999998e-05 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.997555+0000", 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3541999999999999e-05 2024-09-16T04:02:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.997728+0000", 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017327299999999999 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.997760+0000", 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1463e-05 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:00.785 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:58.003638+0000", 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058776100000000001 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:58.003700+0000", 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.2075000000000005e-05 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:00.786 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:00.787 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:00.788 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:00.788 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:00.788 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:00.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:02:00.789 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:00.790 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:00.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:02:00.756+0000 7f3c6dd9b640 0 -- 172.21.15.37:0/3499543067 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7f3c70223b60 msgr2=0x7f3c70225f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:02:00.791 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:02:00.785+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:02:00.892 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:00.889+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:00.916 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:00.916 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:00.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:02:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:set pool 3 pgp_num to 16 2024-09-16T04:02:00.942 INFO:tasks.rados.rados.0.smithi188.stdout:268: done (2 left) 2024-09-16T04:02:00.942 INFO:tasks.rados.rados.0.smithi188.stdout:269: done (1 left) 2024-09-16T04:02:00.942 INFO:tasks.rados.rados.0.smithi188.stdout:270: done (0 left) 2024-09-16T04:02:00.942 INFO:tasks.rados.rados.0.smithi188.stdout:271: rmattr oid 41 current snap is 28 2024-09-16T04:02:00.942 INFO:tasks.rados.rados.0.smithi188.stdout:271: done (0 left) 2024-09-16T04:02:00.942 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 5 current snap is 28 2024-09-16T04:02:00.942 INFO:tasks.rados.rados.0.smithi188.stdout:272: seq_num 116 ranges {1753088=786432,2539520=458752,2998272=442368,3440640=262144} 2024-09-16T04:02:00.957 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:02:00.981 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:00.982 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:00.982 INFO:tasks.rados.rados.0.smithi188.stdout:272: writing smithi18831615-5 from 1753088 to 2539520 tid 1 2024-09-16T04:02:00.990 INFO:tasks.rados.rados.0.smithi188.stdout:272: writing smithi18831615-5 from 2539520 to 2998272 tid 2 2024-09-16T04:02:00.992 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:00.989+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:00.997 INFO:tasks.rados.rados.0.smithi188.stdout:272: writing smithi18831615-5 from 2998272 to 3440640 tid 3 2024-09-16T04:02:01.000 INFO:tasks.rados.rados.0.smithi188.stdout:272: writing smithi18831615-5 from 3440640 to 3702784 tid 4 2024-09-16T04:02:01.001 INFO:tasks.rados.rados.0.smithi188.stdout:273: read oid 17 snap 15 2024-09-16T04:02:01.001 INFO:tasks.rados.rados.0.smithi188.stdout:273: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:02:01.001 INFO:tasks.rados.rados.0.smithi188.stdout:274: copy_from oid 11 from oid 28 current snap is 28 2024-09-16T04:02:01.001 INFO:tasks.rados.rados.0.smithi188.stdout:272: finishing write tid 1 to smithi18831615-5 2024-09-16T04:02:01.001 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 16 current snap is 28 2024-09-16T04:02:01.001 INFO:tasks.rados.rados.0.smithi188.stdout:275: seq_num 117 ranges {2539520=425984,2965504=606208,3571712=540672,4112384=409600} 2024-09-16T04:02:01.009 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:01.009 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:01.030 INFO:tasks.rados.rados.0.smithi188.stdout:275: writing smithi18831615-16 from 2539520 to 2965504 tid 1 2024-09-16T04:02:01.036 INFO:tasks.rados.rados.0.smithi188.stdout:275: writing smithi18831615-16 from 2965504 to 3571712 tid 2 2024-09-16T04:02:01.040 INFO:tasks.rados.rados.0.smithi188.stdout:275: writing smithi18831615-16 from 3571712 to 4112384 tid 3 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:275: writing smithi18831615-16 from 4112384 to 4521984 tid 4 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:276: copy_from oid 21 from oid 49 current snap is 28 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:272: finishing write tid 2 to smithi18831615-5 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:277: snap_create 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:272: finishing write tid 3 to smithi18831615-5 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:272: finishing write tid 4 to smithi18831615-5 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:272: finishing write tid 5 to smithi18831615-5 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:272: finishing write tid 6 to smithi18831615-5 2024-09-16T04:02:01.043 INFO:tasks.rados.rados.0.smithi188.stdout:272: oid 5 updating version 0 to 132 2024-09-16T04:02:01.044 INFO:tasks.rados.rados.0.smithi188.stdout:272: oid 5 version 132 is already newer than 131 2024-09-16T04:02:01.044 INFO:tasks.rados.rados.0.smithi188.stdout:272: oid 5 version 132 is already newer than 128 2024-09-16T04:02:01.044 INFO:tasks.rados.rados.0.smithi188.stdout:272: oid 5 version 132 is already newer than 129 2024-09-16T04:02:01.044 INFO:tasks.rados.rados.0.smithi188.stdout:272: oid 5 version 132 is already newer than 130 2024-09-16T04:02:01.044 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 5 v 132 (ObjNum 116 snap 28 seq_num 116) dirty exists 2024-09-16T04:02:01.044 INFO:tasks.rados.rados.0.smithi188.stdout:272: left oid 5 (ObjNum 116 snap 28 seq_num 116) 2024-09-16T04:02:01.044 INFO:tasks.rados.rados.0.smithi188.stdout:273: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:02:01.064 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:01.064 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:01.064 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:02:01.064 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:02:01.064 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:01.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:02:01.093 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:01.089+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:01.126 INFO:tasks.rados.rados.0.smithi188.stdout:275: finishing write tid 1 to smithi18831615-16 2024-09-16T04:02:01.126 INFO:tasks.rados.rados.0.smithi188.stdout:275: finishing write tid 2 to smithi18831615-16 2024-09-16T04:02:01.126 INFO:tasks.rados.rados.0.smithi188.stdout:274: finishing copy_from racing read to smithi18831615-11 2024-09-16T04:02:01.126 INFO:tasks.rados.rados.0.smithi188.stdout:276: finishing copy_from to smithi18831615-21 2024-09-16T04:02:01.126 INFO:tasks.rados.rados.0.smithi188.stdout:276: got expected ENOENT (src dne) 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:276: finishing copy_from racing read to smithi18831615-21 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:274: finishing copy_from to smithi18831615-11 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 11 v 71 (ObjNum 103 snap 22 seq_num 103) dirty exists 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: finishing write tid 3 to smithi18831615-16 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: finishing write tid 4 to smithi18831615-16 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: finishing write tid 5 to smithi18831615-16 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: finishing write tid 6 to smithi18831615-16 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: oid 16 updating version 0 to 134 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: oid 16 updating version 134 to 137 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: oid 16 version 137 is already newer than 133 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: oid 16 version 137 is already newer than 136 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:275: oid 16 version 137 is already newer than 135 2024-09-16T04:02:01.127 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 16 v 137 (ObjNum 117 snap 28 seq_num 117) dirty exists 2024-09-16T04:02:01.128 INFO:tasks.rados.rados.0.smithi188.stdout:275: left oid 16 (ObjNum 117 snap 28 seq_num 117) 2024-09-16T04:02:01.194 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:01.189+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:01.205 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:7 3.0s0 3:0076079f:::smithi18831615-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.845227+0000", 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 39.355670115000002, 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.017844958000000001, 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.206 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 7 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845227+0000", 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845227+0000", 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.207 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845231+0000", 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.6169999999999998e-06 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845487+0000", 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025666399999999998 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845522+0000", 2024-09-16T04:02:01.208 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4304999999999998e-05 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845543+0000", 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1022999999999998e-05 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.209 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845730+0000", 2024-09-16T04:02:01.210 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00018766 2024-09-16T04:02:01.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.210 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.846074+0000", 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00034338700000000001 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.848021+0000", 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001947442 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.211 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.854006+0000", 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059846090000000001 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.862889+0000", 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0088831780000000003 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.212 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.863072+0000", 2024-09-16T04:02:01.213 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00018307299999999999 2024-09-16T04:02:01.213 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.213 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.213 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:8 3.0s0 3:0076079f:::smithi18831615-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 39.355283968999998, 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.022018801000000001, 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.214 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 8 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:02:01.215 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845615+0000", 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.967e-06 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845638+0000", 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2532e-05 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.216 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845663+0000", 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845680+0000", 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7323e-05 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.851526+0000", 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058460049999999996 2024-09-16T04:02:01.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.851763+0000", 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023643399999999999 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.852890+0000", 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0011268949999999999 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.858042+0000", 2024-09-16T04:02:01.218 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051525850000000003 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.867414+0000", 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093716719999999993 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.867632+0000", 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00021833200000000001 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.219 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:9 3.0s0 3:0076079f:::smithi18831615-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 39.355188716000001, 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.027184505000000001, 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.220 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 9 2024-09-16T04:02:01.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.221 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.221 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.221 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.221 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:02:01.221 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.221 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845710+0000", 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.567e-06 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845718+0000", 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7729999999999999e-06 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845740+0000", 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2048999999999998e-05 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845757+0000", 2024-09-16T04:02:01.223 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7076e-05 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.853731+0000", 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079742270000000004 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.853981+0000", 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025017399999999998 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.224 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.868611+0000", 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014630026000000001 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.868826+0000", 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000214816 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.872893+0000", 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0040667969999999996 2024-09-16T04:02:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:82 3.0s0 3:0baf0dec:::smithi18831615-28:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.902684+0000", 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 39.298212954999997, 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015881954, 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.226 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 82 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.902684+0000", 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.227 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.902684+0000", 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.902687+0000", 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.39e-06 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903496+0000", 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00080918799999999996 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.228 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903528+0000", 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2308999999999998e-05 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903546+0000", 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7776999999999999e-05 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.229 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903687+0000", 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000140815 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.904006+0000", 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00031908799999999998 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.905699+0000", 2024-09-16T04:02:01.230 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001693583 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.913338+0000", 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0076389700000000001 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.918424+0000", 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0050858300000000004 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.231 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.918566+0000", 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00014200399999999999 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:84 3.0s0 3:0baf0dec:::smithi18831615-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 39.297145073000003, 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021781656, 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:01.232 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 84 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:02:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903753+0000", 2024-09-16T04:02:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1039999999999999e-06 2024-09-16T04:02:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903759+0000", 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1380000000000004e-06 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903787+0000", 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7223999999999999e-05 2024-09-16T04:02:01.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.236 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.236 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.236 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903802+0000", 2024-09-16T04:02:01.236 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5301000000000001e-05 2024-09-16T04:02:01.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.236 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.236 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.236 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.908998+0000", 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051958990000000004 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.909182+0000", 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000184143 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.920281+0000", 2024-09-16T04:02:01.237 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011099238000000001 2024-09-16T04:02:01.238 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.238 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.238 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.238 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.920720+0000", 2024-09-16T04:02:01.238 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00043861699999999998 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.925534+0000", 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0048139919999999996 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.240 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:327 3.0s0 3:03f9c209:::smithi18831615-40:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 34.311674394999997, 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.027429133000000001, 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.241 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 327 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889227+0000", 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8700000000000002e-06 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889241+0000", 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4606e-05 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.243 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889275+0000", 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3892e-05 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889293+0000", 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7353000000000001e-05 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.244 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889422+0000", 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000129799 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889435+0000", 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2167000000000001e-05 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.893120+0000", 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003685406 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.245 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.893360+0000", 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023993500000000001 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895575+0000", 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0022147410000000001 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901769+0000", 2024-09-16T04:02:01.246 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061945739999999996 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916453+0000", 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014683732999999999 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.247 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916652+0000", 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019905699999999999 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:328 3.0s0 3:03f9c209:::smithi18831615-40:head [rollback 3] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 34.311563438999997, 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.029259183000000001, 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.248 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 328 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.249 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889335+0000", 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6700000000000002e-07 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.250 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889340+0000", 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8230000000000002e-06 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889352+0000", 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.234e-05 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889360+0000", 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.251 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889591+0000", 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023105599999999999 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889610+0000", 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8749e-05 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.896810+0000", 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0072003049999999997 2024-09-16T04:02:01.252 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.897074+0000", 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00026394400000000001 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898389+0000", 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001314787 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.253 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916676+0000", 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018286922000000001 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918392+0000", 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001716038 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918593+0000", 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000201289 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.254 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.255 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.255 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.255 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.255 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:329 3.0s0 3:03f9c209:::smithi18831615-40:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:01.255 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:02:01.255 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 34.311524016, 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.031812201999999998, 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 329 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:02:01.256 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.257 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.257 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.257 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.257 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:02:01.257 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.257 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.257 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889374+0000", 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0299999999999998e-07 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889377+0000", 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2770000000000001e-06 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.258 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889387+0000", 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6260000000000007e-06 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889394+0000", 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9059999999999996e-06 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889709+0000", 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00031473299999999998 2024-09-16T04:02:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889718+0000", 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2499999999999995e-06 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899428+0000", 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0097098859999999992 2024-09-16T04:02:01.260 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899700+0000", 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000272558 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901104+0000", 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00140375 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920686+0000", 2024-09-16T04:02:01.261 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019581923000000001 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920950+0000", 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00026403800000000002 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921185+0000", 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023545200000000001 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.262 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:330 3.0s0 3:0076079f:::smithi18831615-3:head [stat,delete] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 34.311458098000003, 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.032116411999999997, 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.263 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 330 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.264 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.265 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:02:01.265 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.265 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.265 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.265 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.265 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889440+0000", 2024-09-16T04:02:01.265 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6000000000000003e-07 2024-09-16T04:02:01.265 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.266 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.266 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.266 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889444+0000", 2024-09-16T04:02:01.266 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:02:01.266 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.266 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.266 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.266 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889455+0000", 2024-09-16T04:02:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0259e-05 2024-09-16T04:02:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889462+0000", 2024-09-16T04:02:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4329999999999998e-06 2024-09-16T04:02:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.268 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.268 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889794+0000", 2024-09-16T04:02:01.268 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000331966 2024-09-16T04:02:01.268 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.268 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.268 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.268 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889810+0000", 2024-09-16T04:02:01.268 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5696999999999999e-05 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902422+0000", 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012612490000000001 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.269 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902647+0000", 2024-09-16T04:02:01.270 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00022501300000000001 2024-09-16T04:02:01.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.270 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.270 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.270 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906224+0000", 2024-09-16T04:02:01.270 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0035769059999999999 2024-09-16T04:02:01.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.270 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921192+0000", 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014967644 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921363+0000", 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000171304 2024-09-16T04:02:01.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.272 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.272 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.272 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921556+0000", 2024-09-16T04:02:01.272 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019263700000000001 2024-09-16T04:02:01.272 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.272 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.272 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.273 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.273 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.273 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s4 41/33 ECSubWrite(tid=10, reqid=osd.5.0:10, at_version=41'8, trim_to=0'0, roll_forward_to=37'6))", 2024-09-16T04:02:01.273 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:02:01.273 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 34.297467083000001, 2024-09-16T04:02:01.273 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.033655513999999997, 2024-09-16T04:02:01.273 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.273 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.274 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.275 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:02:01.275 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.275 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.275 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.275 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.275 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903432+0000", 2024-09-16T04:02:01.275 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2970000000000002e-06 2024-09-16T04:02:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903515+0000", 2024-09-16T04:02:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2059999999999997e-05 2024-09-16T04:02:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.277 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903547+0000", 2024-09-16T04:02:01.277 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2106000000000001e-05 2024-09-16T04:02:01.277 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.277 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.277 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.277 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903569+0000", 2024-09-16T04:02:01.277 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1939000000000001e-05 2024-09-16T04:02:01.277 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.278 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.278 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.278 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916774+0000", 2024-09-16T04:02:01.278 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013205099 2024-09-16T04:02:01.278 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.278 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.278 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.278 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916797+0000", 2024-09-16T04:02:01.279 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2809000000000001e-05 2024-09-16T04:02:01.279 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.279 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.279 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.279 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937024+0000", 2024-09-16T04:02:01.279 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020227256999999998 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937086+0000", 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1946999999999995e-05 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:371 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:02:01.280 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 24.482618550000002, 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018868566, 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 371 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.281 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718281+0000", 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1969999999999999e-06 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.282 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718293+0000", 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1636e-05 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718315+0000", 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2935999999999999e-05 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.283 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718327+0000", 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1124000000000001e-05 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718409+0000", 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2245999999999998e-05 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718416+0000", 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.657e-06 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.284 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.724610+0000", 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061949750000000001 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.724979+0000", 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00036804699999999998 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.727047+0000", 2024-09-16T04:02:01.285 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0020681330000000002 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.735227+0000", 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0081803929999999993 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.737066+0000", 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00183858 2024-09-16T04:02:01.286 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.737147+0000", 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1642000000000002e-05 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:372 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [rollback 7] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:02:01.287 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 24.482534201, 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021826534000000002, 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 372 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.288 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718364+0000", 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.289 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718367+0000", 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0489999999999999e-06 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718384+0000", 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7365000000000001e-05 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718398+0000", 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.358e-05 2024-09-16T04:02:01.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718493+0000", 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5422999999999998e-05 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718498+0000", 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7500000000000003e-06 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.728402+0000", 2024-09-16T04:02:01.291 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0099045539999999994 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.728508+0000", 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000105475 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.729094+0000", 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00058619300000000004 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.292 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.736729+0000", 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0076349230000000001 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740059+0000", 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0033296939999999998 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740190+0000", 2024-09-16T04:02:01.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00013097299999999999 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:373 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 24.482486090999998, 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.022197585999999998, 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.294 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 373 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.295 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718412+0000", 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1099999999999995e-07 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718416+0000", 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.873e-06 2024-09-16T04:02:01.296 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718427+0000", 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1233000000000001e-05 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718435+0000", 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7069999999999992e-06 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718521+0000", 2024-09-16T04:02:01.297 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6065999999999999e-05 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718524+0000", 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.72e-06 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732081+0000", 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0135566 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.298 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732182+0000", 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000101362 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732688+0000", 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00050549700000000004 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.736733+0000", 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004045089 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740503+0000", 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003770088 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740609+0000", 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010564 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.300 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:512 3.0s0 3:0baf0dec:::smithi18831615-28:head [copy-from ver 30] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:50.860592+0000", 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.340305735999999, 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.049257191999999998, 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.301 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 512 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860592+0000", 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860592+0000", 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.302 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860594+0000", 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6359999999999998e-06 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860608+0000", 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3366999999999999e-05 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.303 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860640+0000", 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2146e-05 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860657+0000", 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7523000000000001e-05 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860785+0000", 2024-09-16T04:02:01.304 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012776700000000001 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860797+0000", 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.224e-05 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.864255+0000", 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003457781 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.305 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.864407+0000", 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015244999999999999 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.901818+0000", 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.037410153000000002 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.306 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.908299+0000", 2024-09-16T04:02:01.307 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0064817659999999999 2024-09-16T04:02:01.307 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.307 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.307 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:01.307 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.909766+0000", 2024-09-16T04:02:01.307 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0014670900000000001 2024-09-16T04:02:01.308 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.308 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.308 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.308 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.909849+0000", 2024-09-16T04:02:01.308 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2273000000000003e-05 2024-09-16T04:02:01.308 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.309 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.309 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.309 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.309 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.309 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:513 3.0s0 3:0baf0dec:::smithi18831615-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:01.309 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:50.860697+0000", 2024-09-16T04:02:01.310 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.340200032, 2024-09-16T04:02:01.310 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.047167210000000001, 2024-09-16T04:02:01.310 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.310 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.310 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:01.310 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.311 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.311 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.311 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 513 2024-09-16T04:02:01.311 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.311 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.311 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.312 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.312 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860697+0000", 2024-09-16T04:02:01.312 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.312 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.312 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.312 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.313 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860697+0000", 2024-09-16T04:02:01.313 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.313 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.313 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.313 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.314 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860698+0000", 2024-09-16T04:02:01.314 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5999999999999991e-07 2024-09-16T04:02:01.314 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.314 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.314 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.314 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860704+0000", 2024-09-16T04:02:01.314 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.4569999999999996e-06 2024-09-16T04:02:01.315 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.315 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.315 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.315 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860716+0000", 2024-09-16T04:02:01.315 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2823000000000001e-05 2024-09-16T04:02:01.315 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.316 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.316 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.316 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860725+0000", 2024-09-16T04:02:01.316 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.9230000000000008e-06 2024-09-16T04:02:01.316 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.316 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.317 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.317 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860937+0000", 2024-09-16T04:02:01.317 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00021142599999999999 2024-09-16T04:02:01.317 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.317 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.318 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.318 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860956+0000", 2024-09-16T04:02:01.318 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.927e-05 2024-09-16T04:02:01.318 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.318 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.318 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.319 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.864755+0000", 2024-09-16T04:02:01.319 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037985340000000001 2024-09-16T04:02:01.319 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.319 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.319 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:01.319 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.864922+0000", 2024-09-16T04:02:01.320 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016783999999999999 2024-09-16T04:02:01.320 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.320 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.320 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.320 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.907306+0000", 2024-09-16T04:02:01.320 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.042383415000000001 2024-09-16T04:02:01.321 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.321 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.321 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.321 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.907559+0000", 2024-09-16T04:02:01.321 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025296500000000001 2024-09-16T04:02:01.322 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.322 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.322 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.322 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.907864+0000", 2024-09-16T04:02:01.322 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00030559700000000001 2024-09-16T04:02:01.322 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.323 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.323 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.323 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.323 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.323 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:559 3.0s0 3:0baf0dec:::smithi18831615-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:02:01.323 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:52.761892+0000", 2024-09-16T04:02:01.324 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 8.4390050559999992, 2024-09-16T04:02:01.324 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020170703000000002, 2024-09-16T04:02:01.324 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.324 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.324 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:01.324 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.325 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.325 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.325 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 559 2024-09-16T04:02:01.325 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.325 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.326 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.326 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.326 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761892+0000", 2024-09-16T04:02:01.326 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.326 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.326 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.327 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.327 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761892+0000", 2024-09-16T04:02:01.327 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.327 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.327 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.327 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.328 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761893+0000", 2024-09-16T04:02:01.328 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7300000000000005e-07 2024-09-16T04:02:01.328 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.328 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.328 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.329 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761897+0000", 2024-09-16T04:02:01.329 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.534e-06 2024-09-16T04:02:01.329 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.329 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.329 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.330 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761907+0000", 2024-09-16T04:02:01.330 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0317e-05 2024-09-16T04:02:01.330 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.330 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.330 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.330 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761914+0000", 2024-09-16T04:02:01.331 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3119999999999996e-06 2024-09-16T04:02:01.331 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.331 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.331 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.331 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.765546+0000", 2024-09-16T04:02:01.331 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0036314260000000001 2024-09-16T04:02:01.332 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.332 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.332 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:01.332 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.765651+0000", 2024-09-16T04:02:01.333 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010494599999999999 2024-09-16T04:02:01.333 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.333 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.333 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.333 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.778221+0000", 2024-09-16T04:02:01.333 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012570141 2024-09-16T04:02:01.334 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.334 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.334 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.334 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.778348+0000", 2024-09-16T04:02:01.334 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000127494 2024-09-16T04:02:01.335 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.335 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.335 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.335 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.782063+0000", 2024-09-16T04:02:01.335 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037146599999999998 2024-09-16T04:02:01.335 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.336 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.336 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.336 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.336 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.336 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:636 3.9s0 3:9300f872:::smithi18831615-22:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:01.336 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.919425+0000", 2024-09-16T04:02:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.2814727370000001, 2024-09-16T04:02:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018013963000000001, 2024-09-16T04:02:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 636 2024-09-16T04:02:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919425+0000", 2024-09-16T04:02:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919425+0000", 2024-09-16T04:02:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.340 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.340 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.340 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919426+0000", 2024-09-16T04:02:01.340 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7799999999999999e-06 2024-09-16T04:02:01.340 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.341 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.341 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.341 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919435+0000", 2024-09-16T04:02:01.341 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7129999999999994e-06 2024-09-16T04:02:01.341 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.341 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.342 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.342 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919456+0000", 2024-09-16T04:02:01.342 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1157000000000001e-05 2024-09-16T04:02:01.342 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.342 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.342 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.343 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919471+0000", 2024-09-16T04:02:01.343 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4876e-05 2024-09-16T04:02:01.343 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.343 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.343 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.343 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919684+0000", 2024-09-16T04:02:01.344 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000213148 2024-09-16T04:02:01.344 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.344 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.344 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.344 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919701+0000", 2024-09-16T04:02:01.344 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7197000000000002e-05 2024-09-16T04:02:01.345 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.345 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.345 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.345 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.926948+0000", 2024-09-16T04:02:01.345 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0072468239999999998 2024-09-16T04:02:01.346 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.346 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.346 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.346 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.927206+0000", 2024-09-16T04:02:01.346 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025767100000000001 2024-09-16T04:02:01.346 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.347 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.347 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.347 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.937439+0000", 2024-09-16T04:02:01.347 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010232597 2024-09-16T04:02:01.347 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.348 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.348 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.348 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.348 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.348 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:637 3.9s0 3:9f17a056:::smithi18831615-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:01.349 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.919492+0000", 2024-09-16T04:02:01.349 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.281404808, 2024-09-16T04:02:01.349 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019324294999999998, 2024-09-16T04:02:01.349 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.350 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.350 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:01.350 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.350 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.350 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.350 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 637 2024-09-16T04:02:01.351 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.351 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.351 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.351 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.351 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919492+0000", 2024-09-16T04:02:01.351 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.352 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.352 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.352 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.352 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919492+0000", 2024-09-16T04:02:01.352 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.353 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.353 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.353 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.353 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919494+0000", 2024-09-16T04:02:01.353 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1400000000000001e-06 2024-09-16T04:02:01.354 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.354 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.354 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.354 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919529+0000", 2024-09-16T04:02:01.354 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5618999999999999e-05 2024-09-16T04:02:01.354 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.355 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.355 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.355 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919549+0000", 2024-09-16T04:02:01.355 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9565999999999999e-05 2024-09-16T04:02:01.355 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.355 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.355 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.356 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919564+0000", 2024-09-16T04:02:01.356 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.491e-05 2024-09-16T04:02:01.356 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.356 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.357 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.357 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919815+0000", 2024-09-16T04:02:01.357 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025087699999999999 2024-09-16T04:02:01.357 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.357 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.358 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.358 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919827+0000", 2024-09-16T04:02:01.358 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2680000000000001e-05 2024-09-16T04:02:01.358 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.358 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.358 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.359 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.928058+0000", 2024-09-16T04:02:01.359 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0082304679999999995 2024-09-16T04:02:01.359 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.359 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.359 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.359 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.928223+0000", 2024-09-16T04:02:01.359 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016547600000000001 2024-09-16T04:02:01.359 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.938817+0000", 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010593559000000001 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.360 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:638 3.9s0 3:9f17a056:::smithi18831615-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.919585+0000", 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.2813120530000002, 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.023803158000000001, 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 638 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919585+0000", 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.362 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919585+0000", 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919586+0000", 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.363 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919595+0000", 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.3690000000000007e-06 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919613+0000", 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8147000000000001e-05 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.364 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.365 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.365 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919627+0000", 2024-09-16T04:02:01.365 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.432e-05 2024-09-16T04:02:01.365 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.365 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.365 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.365 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919938+0000", 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00031104499999999997 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919956+0000", 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7903000000000001e-05 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.366 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.938944+0000", 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018987456999999999 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.939131+0000", 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00018752800000000001 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.367 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.943388+0000", 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042571960000000004 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.368 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:639 3.9s0 3:9f17a056:::smithi18831615-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.919667+0000", 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.2812306690000002, 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.024857047, 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:01.369 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.370 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 639 2024-09-16T04:02:01.370 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.370 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:01.370 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.370 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:01.370 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919667+0000", 2024-09-16T04:02:01.371 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.371 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.371 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.371 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:01.371 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919667+0000", 2024-09-16T04:02:01.372 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:01.372 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.372 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.372 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:01.372 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919668+0000", 2024-09-16T04:02:01.372 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2270000000000001e-06 2024-09-16T04:02:01.373 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.373 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.373 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:01.373 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919675+0000", 2024-09-16T04:02:01.373 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.5399999999999998e-06 2024-09-16T04:02:01.373 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.374 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.374 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:01.374 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919695+0000", 2024-09-16T04:02:01.374 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0106e-05 2024-09-16T04:02:01.374 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.375 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.375 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.375 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919711+0000", 2024-09-16T04:02:01.375 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5265999999999999e-05 2024-09-16T04:02:01.376 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.376 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.376 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.376 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.920054+0000", 2024-09-16T04:02:01.376 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000343655 2024-09-16T04:02:01.377 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.377 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.377 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.377 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.920071+0000", 2024-09-16T04:02:01.377 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6852999999999999e-05 2024-09-16T04:02:01.377 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.378 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.378 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:01.378 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.939740+0000", 2024-09-16T04:02:01.378 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019668486999999998 2024-09-16T04:02:01.378 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.379 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.379 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:01.379 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.940021+0000", 2024-09-16T04:02:01.379 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00028088700000000001 2024-09-16T04:02:01.379 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:01.379 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:01.379 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:01.380 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.944524+0000", 2024-09-16T04:02:01.380 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0045030260000000003 2024-09-16T04:02:01.380 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.380 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.380 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.380 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:01.380 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:01.381 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:01.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:02:01.384 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:01.381+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:01.483 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:01.481+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:01.514 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:01.515 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:01.515 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:02:01.515 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:01.528 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:02:01.584 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:01.581+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:01.650 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:01.650 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:01.650 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:02:01.650 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:02:01.650 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:01.664 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:02:01.711 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:01.709+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:01.724 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-09-16T04:02:01.731 INFO:tasks.rados.rados.0.smithi188.stdout:272: done (5 left) 2024-09-16T04:02:01.731 INFO:tasks.rados.rados.0.smithi188.stdout:273: done (4 left) 2024-09-16T04:02:01.732 INFO:tasks.rados.rados.0.smithi188.stdout:274: done (3 left) 2024-09-16T04:02:01.732 INFO:tasks.rados.rados.0.smithi188.stdout:275: done (2 left) 2024-09-16T04:02:01.732 INFO:tasks.rados.rados.0.smithi188.stdout:276: done (1 left) 2024-09-16T04:02:01.732 INFO:tasks.rados.rados.0.smithi188.stdout:277: done (0 left) 2024-09-16T04:02:01.732 INFO:tasks.rados.rados.0.smithi188.stdout:278: copy_from oid 34 from oid 49 current snap is 29 2024-09-16T04:02:01.732 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 4 current snap is 29 2024-09-16T04:02:01.732 INFO:tasks.rados.rados.0.smithi188.stdout:279: seq_num 118 ranges {1376256=442368,1818624=753664,2572288=655360,3227648=147456} 2024-09-16T04:02:01.743 INFO:tasks.rados.rados.0.smithi188.stdout:279: writing smithi18831615-4 from 1376256 to 1818624 tid 1 2024-09-16T04:02:01.749 INFO:tasks.rados.rados.0.smithi188.stdout:279: writing smithi18831615-4 from 1818624 to 2572288 tid 2 2024-09-16T04:02:01.753 INFO:tasks.rados.rados.0.smithi188.stdout:279: writing smithi18831615-4 from 2572288 to 3227648 tid 3 2024-09-16T04:02:01.755 INFO:tasks.rados.rados.0.smithi188.stdout:279: writing smithi18831615-4 from 3227648 to 3375104 tid 4 2024-09-16T04:02:01.755 INFO:tasks.rados.rados.0.smithi188.stdout:280: delete oid 28 current snap is 29 2024-09-16T04:02:01.755 INFO:tasks.rados.rados.0.smithi188.stdout:278: finishing copy_from to smithi18831615-34 2024-09-16T04:02:01.755 INFO:tasks.rados.rados.0.smithi188.stdout:278: got expected ENOENT (src dne) 2024-09-16T04:02:01.755 INFO:tasks.rados.rados.0.smithi188.stdout:278: finishing copy_from racing read to smithi18831615-34 2024-09-16T04:02:01.764 INFO:tasks.rados.rados.0.smithi188.stdout:279: finishing write tid 1 to smithi18831615-4 2024-09-16T04:02:01.764 INFO:tasks.rados.rados.0.smithi188.stdout:279: finishing write tid 2 to smithi18831615-4 2024-09-16T04:02:01.764 INFO:tasks.rados.rados.0.smithi188.stdout:279: finishing write tid 3 to smithi18831615-4 2024-09-16T04:02:01.766 INFO:tasks.rados.rados.0.smithi188.stdout:279: finishing write tid 4 to smithi18831615-4 2024-09-16T04:02:01.766 INFO:tasks.rados.rados.0.smithi188.stdout:279: finishing write tid 5 to smithi18831615-4 2024-09-16T04:02:01.766 INFO:tasks.rados.rados.0.smithi188.stdout:278: done (2 left) 2024-09-16T04:02:01.766 INFO:tasks.rados.rados.0.smithi188.stdout:280: done (1 left) 2024-09-16T04:02:01.766 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 7 current snap is 29 2024-09-16T04:02:01.766 INFO:tasks.rados.rados.0.smithi188.stdout:281: seq_num 120 ranges {819200=475136,1294336=655360,1949696=622592,2572288=278528} 2024-09-16T04:02:01.774 INFO:tasks.rados.rados.0.smithi188.stdout:281: writing smithi18831615-7 from 819200 to 1294336 tid 1 2024-09-16T04:02:01.779 INFO:tasks.rados.rados.0.smithi188.stdout:281: writing smithi18831615-7 from 1294336 to 1949696 tid 2 2024-09-16T04:02:01.784 INFO:tasks.rados.rados.0.smithi188.stdout:281: writing smithi18831615-7 from 1949696 to 2572288 tid 3 2024-09-16T04:02:01.785 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:01.781+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:01.786 INFO:tasks.rados.rados.0.smithi188.stdout:281: writing smithi18831615-7 from 2572288 to 2850816 tid 4 2024-09-16T04:02:01.786 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 24 current snap is 29 2024-09-16T04:02:01.786 INFO:tasks.rados.rados.0.smithi188.stdout:282: seq_num 121 ranges {1572864=491520,2064384=802816,2867200=720896,3588096=32768} 2024-09-16T04:02:01.800 INFO:tasks.rados.rados.0.smithi188.stdout:282: writing smithi18831615-24 from 1572864 to 2064384 tid 1 2024-09-16T04:02:01.801 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4371.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:07.735249+0000", 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 54.062996120000001, 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074231499999999999, 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4371", 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.37:0/2503621667", 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735249+0000", 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735249+0000", 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735251+0000", 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6599999999999999e-06 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735265+0000", 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3862e-05 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735297+0000", 2024-09-16T04:02:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1516000000000001e-05 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735313+0000", 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.666e-05 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735477+0000", 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016360100000000001 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.735495+0000", 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8416000000000001e-05 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.739219+0000", 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037237609999999999 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.739432+0000", 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021331 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 12", 2024-09-16T04:02:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.739697+0000", 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000265104 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.741578+0000", 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001880844 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.742529+0000", 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000950661 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.742607+0000", 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8196999999999997e-05 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:07.742672+0000", 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4371.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-16T04:02:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:08.740538+0000", 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 53.057707022000002, 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073770179999999999, 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4371", 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.37:0/2503621667", 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740538+0000", 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740538+0000", 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740541+0000", 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0469999999999998e-06 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740555+0000", 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4195999999999999e-05 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740586+0000", 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1195000000000002e-05 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740601+0000", 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5330000000000001e-05 2024-09-16T04:02:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740778+0000", 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017643100000000001 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.740795+0000", 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7102e-05 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.744656+0000", 2024-09-16T04:02:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038614270000000002 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.744835+0000", 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017891000000000001 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 12", 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.745081+0000", 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024605599999999998 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-09-16T04:02:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.746669+0000", 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001587209 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.747763+0000", 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010943859999999999 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.747847+0000", 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4481000000000007e-05 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:08.747915+0000", 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7248000000000004e-05 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s2 40/33 ECSubWrite(tid=191, reqid=client.4646.0:318, at_version=40'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.028860+0000", 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 35.769384162000001, 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073691670000000003, 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028860+0000", 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028860+0000", 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028861+0000", 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8700000000000005e-07 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029461+0000", 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060003400000000005 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029496+0000", 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4965999999999997e-05 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029523+0000", 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7212e-05 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029703+0000", 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017971600000000001 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029726+0000", 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2816999999999999e-05 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.036188+0000", 2024-09-16T04:02:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064616860000000003 2024-09-16T04:02:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.036230+0000", 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1949000000000001e-05 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:349 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:29.923486+0000", 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.874758224000001, 2024-09-16T04:02:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081109709999999998, 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 349 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923486+0000", 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923486+0000", 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923489+0000", 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6309999999999999e-06 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923502+0000", 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3203e-05 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923527+0000", 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4270999999999999e-05 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923538+0000", 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1623999999999999e-05 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923713+0000", 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017450699999999999 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923728+0000", 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5383000000000002e-05 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.926449+0000", 2024-09-16T04:02:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002720779 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.926529+0000", 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9796000000000006e-05 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.931597+0000", 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005068777 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:350 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.874678573000001, 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092273240000000003, 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 350 2024-09-16T04:02:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923567+0000", 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7e-07 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923571+0000", 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2830000000000003e-06 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923578+0000", 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2810000000000003e-06 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923582+0000", 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1500000000000001e-06 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923849+0000", 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000266464 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923854+0000", 2024-09-16T04:02:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.989e-06 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927191+0000", 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033374810000000002 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927250+0000", 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8369000000000003e-05 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.932793+0000", 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055438370000000002 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:351 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:02:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.874596318999998, 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097430910000000006, 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 351 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923649+0000", 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2399999999999999e-07 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923652+0000", 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.827e-06 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923659+0000", 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6420000000000001e-06 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923663+0000", 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7900000000000001e-06 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923914+0000", 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000251387 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923926+0000", 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1933000000000001e-05 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927543+0000", 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036171580000000001 2024-09-16T04:02:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927607+0000", 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3925999999999997e-05 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.933391+0000", 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005784104 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 51/33 ECSubWrite(tid=75, reqid=client.4646.0:384, at_version=51'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.945726358999998, 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099262940000000004, 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852519+0000", 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000004e-07 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853510+0000", 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099128100000000011 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853546+0000", 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5522000000000003e-05 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853572+0000", 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5924000000000001e-05 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853747+0000", 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017537299999999999 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853774+0000", 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7129000000000001e-05 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.862371+0000", 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085965390000000003 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.862445+0000", 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3942000000000005e-05 2024-09-16T04:02:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 51/33 ECSubWrite(tid=78, reqid=client.4646.0:387, at_version=51'9, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.860068+0000", 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.938176773999999, 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051979000000000001, 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860068+0000", 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860068+0000", 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860068+0000", 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860085+0000", 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6691999999999999e-05 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860096+0000", 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0421999999999999e-05 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.860103+0000", 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5800000000000003e-06 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.861790+0000", 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001687137 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.861820+0000", 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9269e-05 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.865210+0000", 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00339048 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.865266+0000", 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5739000000000003e-05 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 54/33 ECSubWrite(tid=34, reqid=client.4646.0:420, at_version=54'24, trim_to=0'0, roll_forward_to=42'21))", 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.809685+0000", 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.988559767000002, 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052442230000000001, 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.809685+0000", 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.809685+0000", 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.809687+0000", 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.283e-06 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810192+0000", 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050483799999999997 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810225+0000", 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3383000000000002e-05 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810243+0000", 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7472000000000001e-05 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810434+0000", 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000191194 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.810463+0000", 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8728999999999999e-05 2024-09-16T04:02:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814856+0000", 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043929629999999997 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814929+0000", 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3361000000000005e-05 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 54/33 ECSubWrite(tid=211, reqid=osd.13.0:211, at_version=54'19, trim_to=0'0, roll_forward_to=53'17))", 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.932317+0000", 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.865927382999999, 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050579199999999996, 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932317+0000", 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932317+0000", 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932318+0000", 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9000000000000002e-07 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932433+0000", 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011560799999999999 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932464+0000", 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0602999999999998e-05 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.932487+0000", 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3456e-05 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.933037+0000", 2024-09-16T04:02:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054949000000000003 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.933065+0000", 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8589999999999999e-05 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.937292+0000", 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042269109999999999 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.937375+0000", 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2872000000000002e-05 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 56/33 ECSubWrite(tid=93, reqid=osd.14.0:91, at_version=0'0, trim_to=0'0, roll_forward_to=56'15))", 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.735189+0000", 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.063056053, 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051882819999999998, 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735189+0000", 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735189+0000", 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735190+0000", 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8500000000000001e-06 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735239+0000", 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8202999999999999e-05 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735264+0000", 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5412e-05 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735287+0000", 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2626999999999999e-05 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735440+0000", 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015364999999999999 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.735466+0000", 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5786999999999999e-05 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.740251+0000", 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047844879999999999 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.740377+0000", 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000126265 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 56/33 ECSubWrite(tid=213, reqid=client.4646.0:430, at_version=56'21, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.757374+0000", 2024-09-16T04:02:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.040870510000001, 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051080140000000001, 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757374+0000", 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757374+0000", 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.757375+0000", 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7700000000000004e-07 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758147+0000", 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077236600000000002 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758182+0000", 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5040000000000003e-05 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758206+0000", 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3399000000000001e-05 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758401+0000", 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019587700000000001 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.758434+0000", 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2252000000000002e-05 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.762394+0000", 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039598250000000002 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.762482+0000", 2024-09-16T04:02:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8578000000000005e-05 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 66/33 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=66'32, trim_to=0'0, roll_forward_to=63'30))", 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.737986+0000", 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.060259064, 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065142179999999996, 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737986+0000", 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737986+0000", 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737986+0000", 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5599999999999995e-07 2024-09-16T04:02:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.738033+0000", 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6490999999999999e-05 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.738057+0000", 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4442000000000001e-05 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.738080+0000", 2024-09-16T04:02:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.738307+0000", 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022662999999999999 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.738351+0000", 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4014000000000002e-05 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.744411+0000", 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060598639999999999 2024-09-16T04:02:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.744500+0000", 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8999000000000006e-05 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 67/33 ECSubWrite(tid=106, reqid=client.4646.0:560, at_version=67'17, trim_to=0'0, roll_forward_to=56'15))", 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:52.764835+0000", 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0334095080000001, 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0062787269999999996, 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.764835+0000", 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.764835+0000", 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.764836+0000", 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1299999999999999e-07 2024-09-16T04:02:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.764896+0000", 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0154e-05 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.764940+0000", 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4214000000000001e-05 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.764967+0000", 2024-09-16T04:02:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6435e-05 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.765174+0000", 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020681200000000001 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.765201+0000", 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7375000000000001e-05 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.771032+0000", 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058308149999999996 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.771114+0000", 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2108999999999996e-05 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 67/33 ECSubWrite(tid=108, reqid=client.4646.0:561, at_version=67'18, trim_to=0'0, roll_forward_to=56'15))", 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:52.767128+0000", 2024-09-16T04:02:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0311163990000001, 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050213610000000002, 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767128+0000", 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767128+0000", 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767130+0000", 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9099999999999999e-06 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767183+0000", 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2602999999999998e-05 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767218+0000", 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5188000000000003e-05 2024-09-16T04:02:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767241+0000", 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3519e-05 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767920+0000", 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067860200000000005 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767950+0000", 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9862e-05 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.772057+0000", 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041074960000000004 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.772150+0000", 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2181000000000004e-05 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 70/33 ECSubWrite(tid=247, reqid=client.4646.0:612, at_version=70'42, trim_to=0'0, roll_forward_to=69'40))", 2024-09-16T04:02:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.926381+0000", 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.8718637989999998, 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0062191390000000003, 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.926381+0000", 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.926381+0000", 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.926382+0000", 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.927002+0000", 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000620661 2024-09-16T04:02:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.927047+0000", 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4660999999999998e-05 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.927076+0000", 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9374999999999999e-05 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.927258+0000", 2024-09-16T04:02:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000181807 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.927284+0000", 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6112e-05 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.932560+0000", 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052755450000000004 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.932600+0000", 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0258e-05 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:620 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928440+0000", 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.8698045649999999, 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020208812, 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 620 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928440+0000", 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928440+0000", 2024-09-16T04:02:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928443+0000", 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.463e-06 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928454+0000", 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1259e-05 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928474+0000", 2024-09-16T04:02:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0698000000000001e-05 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928490+0000", 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5546e-05 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928672+0000", 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018192600000000001 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928682+0000", 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0492000000000001e-05 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.930911+0000", 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022290159999999999 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.931035+0000", 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000123149 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.948649+0000", 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017614263000000002 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:621 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928525+0000", 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.8697192620000003, 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020965587000000001, 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 621 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928525+0000", 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928525+0000", 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928526+0000", 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3200000000000003e-07 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928533+0000", 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928544+0000", 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0975000000000001e-05 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928553+0000", 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9050000000000003e-06 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928920+0000", 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036759500000000002 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928947+0000", 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6876000000000002e-05 2024-09-16T04:02:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.940657+0000", 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011709913000000001 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.940806+0000", 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014908599999999999 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.949491+0000", 2024-09-16T04:02:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086846979999999994 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:622 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928566+0000", 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.8696782230000002, 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021638297000000001, 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 622 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928566+0000", 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928566+0000", 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928567+0000", 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4600000000000004e-07 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928573+0000", 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6799999999999998e-06 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928583+0000", 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7389999999999994e-06 2024-09-16T04:02:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928593+0000", 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9299999999999998e-06 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.929017+0000", 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042487699999999999 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.929028+0000", 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0302000000000001e-05 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941680+0000", 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012652029 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941790+0000", 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110104 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.950205+0000", 2024-09-16T04:02:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0084148899999999995 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpReadReply(3.as0 70/33 ECSubReadReply(tid=20, attrs_read=0))", 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.932357+0000", 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.8658875090000002, 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0088944809999999992, 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.932357+0000", 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.932357+0000", 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.932358+0000", 2024-09-16T04:02:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0189999999999999e-06 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.934001+0000", 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00164285 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.934024+0000", 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3436000000000001e-05 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.934041+0000", 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6909e-05 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941200+0000", 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071581789999999998 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941252+0000", 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2088e-05 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:01.899 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:282: writing smithi18831615-24 from 2064384 to 2867200 tid 2 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:282: writing smithi18831615-24 from 2867200 to 3588096 tid 3 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:282: writing smithi18831615-24 from 3588096 to 3620864 tid 4 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:283: setattr oid 5 current snap is 29 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:279: finishing write tid 6 to smithi18831615-4 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:279: oid 4 updating version 0 to 169 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:279: oid 4 updating version 169 to 172 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:279: oid 4 version 172 is already newer than 171 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:279: oid 4 version 172 is already newer than 168 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:279: oid 4 version 172 is already newer than 170 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 4 v 172 (ObjNum 118 snap 29 seq_num 118) dirty exists 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:279: left oid 4 (ObjNum 118 snap 29 seq_num 118) 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:281: finishing write tid 1 to smithi18831615-7 2024-09-16T04:02:01.901 INFO:tasks.rados.rados.0.smithi188.stdout:281: finishing write tid 2 to smithi18831615-7 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: finishing write tid 3 to smithi18831615-7 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: finishing write tid 4 to smithi18831615-7 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: finishing write tid 5 to smithi18831615-7 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: finishing write tid 6 to smithi18831615-7 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: oid 7 updating version 0 to 59 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: oid 7 version 59 is already newer than 58 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: oid 7 updating version 59 to 60 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: oid 7 version 60 is already newer than 56 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: oid 7 version 60 is already newer than 57 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 7 v 60 (ObjNum 120 snap 29 seq_num 120) dirty exists 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: left oid 7 (ObjNum 120 snap 29 seq_num 120) 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:282: finishing write tid 1 to smithi18831615-24 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:279: done (3 left) 2024-09-16T04:02:01.902 INFO:tasks.rados.rados.0.smithi188.stdout:281: done (2 left) 2024-09-16T04:02:01.903 INFO:tasks.rados.rados.0.smithi188.stdout:284: copy_from oid 18 from oid 17 current snap is 29 2024-09-16T04:02:01.903 INFO:tasks.rados.rados.0.smithi188.stdout:285: copy_from oid 50 from oid 14 current snap is 29 2024-09-16T04:02:01.903 INFO:tasks.rados.rados.0.smithi188.stdout:286: read oid 47 snap 25 2024-09-16T04:02:01.903 INFO:tasks.rados.rados.0.smithi188.stdout:286: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:02:01.903 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 48 current snap is 29 2024-09-16T04:02:01.903 INFO:tasks.rados.rados.0.smithi188.stdout:287: seq_num 122 ranges {2211840=507904,2719744=425984,3145728=442368,3588096=540672,4128768=147456} 2024-09-16T04:02:01.904 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:02:01.905 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:01.897+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:287: writing smithi18831615-48 from 2211840 to 2719744 tid 1 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:287: writing smithi18831615-48 from 2719744 to 3145728 tid 2 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:287: writing smithi18831615-48 from 3145728 to 3588096 tid 3 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:287: writing smithi18831615-48 from 3588096 to 4128768 tid 4 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:287: writing smithi18831615-48 from 4128768 to 4276224 tid 5 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:288: delete oid 2 current snap is 29 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:285: finishing copy_from to smithi18831615-50 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:285: got expected ENOENT (src dne) 2024-09-16T04:02:01.906 INFO:tasks.rados.rados.0.smithi188.stdout:285: finishing copy_from racing read to smithi18831615-50 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: finishing write tid 2 to smithi18831615-24 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:286: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 5 v 138 (ObjNum 116 snap 28 seq_num 116) dirty exists 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: finishing write tid 3 to smithi18831615-24 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: finishing write tid 4 to smithi18831615-24 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: finishing write tid 5 to smithi18831615-24 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: finishing write tid 6 to smithi18831615-24 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: oid 24 updating version 0 to 176 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: oid 24 version 176 is already newer than 175 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: oid 24 updating version 176 to 177 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: oid 24 version 177 is already newer than 174 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: oid 24 updating version 177 to 178 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 24 v 178 (ObjNum 121 snap 29 seq_num 121) dirty exists 2024-09-16T04:02:01.907 INFO:tasks.rados.rados.0.smithi188.stdout:282: left oid 24 (ObjNum 121 snap 29 seq_num 121) 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:282: done (6 left) 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:283: done (5 left) 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:285: done (4 left) 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:286: done (3 left) 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:288: done (2 left) 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:289: rmattr oid 42 current snap is 29 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:289: done (2 left) 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:284: finishing copy_from racing read to smithi18831615-18 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:287: finishing write tid 1 to smithi18831615-48 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:284: finishing copy_from to smithi18831615-18 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 18 v 16 (ObjNum 76 snap 3 seq_num 76) dirty exists 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:287: finishing write tid 2 to smithi18831615-48 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:287: finishing write tid 3 to smithi18831615-48 2024-09-16T04:02:01.908 INFO:tasks.rados.rados.0.smithi188.stdout:287: finishing write tid 4 to smithi18831615-48 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: finishing write tid 5 to smithi18831615-48 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: finishing write tid 6 to smithi18831615-48 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: finishing write tid 7 to smithi18831615-48 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: oid 48 updating version 0 to 183 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: oid 48 version 183 is already newer than 182 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: oid 48 version 183 is already newer than 181 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: oid 48 updating version 183 to 184 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: oid 48 updating version 184 to 185 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: oid 48 version 185 is already newer than 180 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 48 v 185 (ObjNum 122 snap 29 seq_num 122) dirty exists 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: left oid 48 (ObjNum 122 snap 29 seq_num 122) 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:290: read oid 9 snap 5 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:290: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:284: done (2 left) 2024-09-16T04:02:01.909 INFO:tasks.rados.rados.0.smithi188.stdout:287: done (1 left) 2024-09-16T04:02:01.910 INFO:tasks.rados.rados.0.smithi188.stdout:291: read oid 49 snap 5 2024-09-16T04:02:01.910 INFO:tasks.rados.rados.0.smithi188.stdout:291: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T04:02:01.910 INFO:tasks.rados.rados.0.smithi188.stdout:292: delete oid 46 current snap is 29 2024-09-16T04:02:01.910 INFO:tasks.rados.rados.0.smithi188.stdout:290: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:02:01.910 INFO:tasks.rados.rados.0.smithi188.stdout:291: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T04:02:01.943 INFO:tasks.rados.rados.0.smithi188.stdout:290: done (2 left) 2024-09-16T04:02:01.943 INFO:tasks.rados.rados.0.smithi188.stdout:291: done (1 left) 2024-09-16T04:02:01.943 INFO:tasks.rados.rados.0.smithi188.stdout:292: done (0 left) 2024-09-16T04:02:01.943 INFO:tasks.rados.rados.0.smithi188.stdout:293: copy_from oid 35 from oid 19 current snap is 29 2024-09-16T04:02:01.943 INFO:tasks.rados.rados.0.smithi188.stdout:294: read oid 25 snap -1 2024-09-16T04:02:01.943 INFO:tasks.rados.rados.0.smithi188.stdout:294: expect (ObjNum 97 snap 19 seq_num 97) 2024-09-16T04:02:01.943 INFO:tasks.rados.rados.0.smithi188.stdout:295: snap_remove snap 14 2024-09-16T04:02:02.001 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:01.997+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:02.071 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:02.071 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:02.071 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:02:02.072 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:02.085 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:02:02.102 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:02.101+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:02.203 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:02.201+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:02.207 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:02.207 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:02.207 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:02:02.207 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:02:02.207 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:02.222 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:02:02.304 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:02.301+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:02.348 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:02.348 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:02:02.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:02:02.348 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:02:02.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.348 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:419 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~671744] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.551665029999999, 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019325796999999999, 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 419 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792510+0000", 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5359999999999999e-06 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793316+0000", 2024-09-16T04:02:02.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00080598599999999996 2024-09-16T04:02:02.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793351+0000", 2024-09-16T04:02:02.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4473e-05 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793368+0000", 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6943e-05 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793504+0000", 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013629000000000001 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793707+0000", 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020315399999999999 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796655+0000", 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002948301 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.800082+0000", 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003426999 2024-09-16T04:02:02.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.811562+0000", 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011479896999999999 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.811833+0000", 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027121799999999999 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:420 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~770048] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.805716+0000", 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.538456874000001, 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014453887, 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 420 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.805716+0000", 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.805716+0000", 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.805720+0000", 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0130000000000004e-06 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806805+0000", 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001085014 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806848+0000", 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2972999999999997e-05 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806871+0000", 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3206000000000001e-05 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807049+0000", 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017784699999999999 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807342+0000", 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029323499999999998 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.809728+0000", 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023861239999999999 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814729+0000", 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050004710000000003 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.819977+0000", 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052487189999999998 2024-09-16T04:02:02.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.820170+0000", 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019228499999999999 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:421 3.ds0 3:bab6d723:::smithi18831615-7:head [setxattr _header (56) in=7b] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.806948+0000", 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.537224983000002, 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014216368, 2024-09-16T04:02:02.360 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 421 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806948+0000", 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806948+0000", 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806950+0000", 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.894e-06 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806960+0000", 2024-09-16T04:02:02.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0756e-05 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806982+0000", 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1993000000000001e-05 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.806999+0000", 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6835999999999999e-05 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.811960+0000", 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004960402 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.812241+0000", 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028163899999999998 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.813544+0000", 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013022999999999999 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.817454+0000", 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039106820000000004 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.820833+0000", 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033792119999999999 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.821164+0000", 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000330654 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.365 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:422 3.ds0 3:bab6d723:::smithi18831615-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.537146901, 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02126656, 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.366 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 422 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807027+0000", 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3969999999999999e-06 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807034+0000", 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1459999999999999e-06 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807055+0000", 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.09e-05 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807070+0000", 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4932999999999999e-05 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814329+0000", 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007259138 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814519+0000", 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000189435 2024-09-16T04:02:02.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.822102+0000", 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075835199999999998 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.822353+0000", 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025044999999999999 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.828292+0000", 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059396409999999998 2024-09-16T04:02:02.370 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:427 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.621975450000001, 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018266843000000001, 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.371 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 427 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722200+0000", 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6929999999999999e-06 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722212+0000", 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2577e-05 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722245+0000", 2024-09-16T04:02:02.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2162999999999997e-05 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722262+0000", 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6843000000000001e-05 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722406+0000", 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000144668 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:02.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722421+0000", 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5271999999999999e-05 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.728611+0000", 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006189193 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.728887+0000", 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027599500000000002 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.740464+0000", 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011577439 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:428 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.621873438000001, 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019155492999999999, 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.376 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 428 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722300+0000", 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7999999999999993e-07 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722308+0000", 2024-09-16T04:02:02.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8259999999999993e-06 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722350+0000", 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1730000000000002e-05 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722359+0000", 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5289999999999997e-06 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722581+0000", 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000221623 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722605+0000", 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4116000000000001e-05 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730680+0000", 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080751769999999994 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730940+0000", 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000259382 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741455+0000", 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010515129999999999 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.381 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:429 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.621796573000001, 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024602057, 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 429 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722377+0000", 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722382+0000", 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7770000000000002e-06 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722392+0000", 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0020000000000001e-05 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722399+0000", 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4000000000000003e-06 2024-09-16T04:02:02.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722655+0000", 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025568500000000002 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722664+0000", 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4129999999999995e-06 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741688+0000", 2024-09-16T04:02:02.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019023476000000001 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741991+0000", 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030357799999999999 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.746978+0000", 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049869179999999999 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:520 3.ds0 3:b4418190:::smithi18831615-35:head [read 0~1687552,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861514+0000", 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.482658677, 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017503623999999999, 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 520 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.387 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861514+0000", 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861514+0000", 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861518+0000", 2024-09-16T04:02:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5930000000000002e-06 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861534+0000", 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6864999999999999e-05 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861589+0000", 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4596999999999999e-05 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861620+0000", 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1126e-05 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861806+0000", 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018558499999999999 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861825+0000", 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9629000000000001e-05 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.866477+0000", 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046518109999999996 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.866743+0000", 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000265889 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.879018+0000", 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012274528999999999 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.391 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:521 3.ds0 3:b4418190:::smithi18831615-35:head [read 0~1687552,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861690+0000", 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.482482940000001, 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022625292000000002, 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 521 2024-09-16T04:02:02.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861690+0000", 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861690+0000", 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861691+0000", 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.753e-06 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861702+0000", 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0451999999999999e-05 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861729+0000", 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6988999999999999e-05 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861746+0000", 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7176e-05 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.862087+0000", 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000340584 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.862110+0000", 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3281999999999998e-05 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.869108+0000", 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0069983629999999996 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.869323+0000", 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021468100000000001 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.884315+0000", 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992012000000001 2024-09-16T04:02:02.396 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:522 3.ds0 3:b4418190:::smithi18831615-35:head [read 0~1687552,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861797+0000", 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.482375436, 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027488874999999999, 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.397 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 522 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861797+0000", 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861797+0000", 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861799+0000", 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.44e-06 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861809+0000", 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0424999999999999e-05 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861831+0000", 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2345999999999999e-05 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861847+0000", 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5461999999999999e-05 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.862212+0000", 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036481699999999997 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.862233+0000", 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1257999999999999e-05 2024-09-16T04:02:02.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.871538+0000", 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093055050000000004 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.871820+0000", 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000281351 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.889286+0000", 2024-09-16T04:02:02.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017466270999999998 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpReadReply(3.ds0 64/33 ECSubReadReply(tid=52, attrs_read=0))", 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.868989+0000", 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.475183244, 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010299043000000001, 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.868989+0000", 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.868989+0000", 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.868991+0000", 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.823e-06 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.872665+0000", 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036738809999999999 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.872923+0000", 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025786299999999999 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.872941+0000", 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8386999999999999e-05 2024-09-16T04:02:02.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.873940+0000", 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099873699999999998 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.879288+0000", 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053483519999999998 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.405 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpReadReply(3.ds0 64/33 ECSubReadReply(tid=53, attrs_read=0))", 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.874912+0000", 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.469260351000001, 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0095312590000000003, 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.874912+0000", 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.874912+0000", 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.874914+0000", 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9740000000000001e-06 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.875553+0000", 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063823799999999996 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.875590+0000", 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7336999999999998e-05 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.875638+0000", 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8488000000000001e-05 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.880773+0000", 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051341470000000004 2024-09-16T04:02:02.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.884444+0000", 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036710750000000002 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpReadReply(3.ds0 65/33 ECSubReadReply(tid=54, attrs_read=0))", 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.875724+0000", 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.468448612, 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013730462000000001, 2024-09-16T04:02:02.409 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.875724+0000", 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.875724+0000", 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.875725+0000", 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.113e-06 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.876634+0000", 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090834900000000003 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.876657+0000", 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3112000000000002e-05 2024-09-16T04:02:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.876740+0000", 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3149e-05 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.886897+0000", 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010157223999999999 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.889455+0000", 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025575149999999998 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 66/33 ECSubWrite(tid=140, reqid=osd.4.0:140, at_version=66'67, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.729435+0000", 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.614738036, 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014463760000000001, 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:02.413 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.729435+0000", 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.729435+0000", 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.729435+0000", 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:02:02.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.729502+0000", 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6746000000000001e-05 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.729544+0000", 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2211e-05 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.729570+0000", 2024-09-16T04:02:02.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5531999999999999e-05 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.729760+0000", 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019006900000000001 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.729788+0000", 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7648e-05 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.743792+0000", 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014004367 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.743898+0000", 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010649000000000001 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.bs5 66/33 ECSubWrite(tid=141, reqid=osd.4.0:141, at_version=66'69, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.731594+0000", 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.612578783, 2024-09-16T04:02:02.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013368998999999999, 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.731594+0000", 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.731594+0000", 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.731596+0000", 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1430000000000001e-06 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.731648+0000", 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2139999999999999e-05 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.731682+0000", 2024-09-16T04:02:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3717999999999998e-05 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.731705+0000", 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3076000000000001e-05 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.732472+0000", 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076718200000000004 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.732496+0000", 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3572000000000001e-05 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.744884+0000", 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012388689 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.744963+0000", 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8479000000000005e-05 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(osd.11.22:2 3.ds0 3:bab6d723:::smithi18831615-7:head [assert-version v51,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.009126+0000", 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3350463789999996, 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030180828, 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009126+0000", 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009126+0000", 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009129+0000", 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9830000000000001e-06 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009141+0000", 2024-09-16T04:02:02.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.163e-05 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009162+0000", 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1299000000000001e-05 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009177+0000", 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5031999999999999e-05 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009257+0000", 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0247999999999995e-05 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009338+0000", 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0375000000000002e-05 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.039307+0000", 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029969261000000001 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.425 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:726 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~475136] snapc 1e=[1e,1d,1c,1b,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.774937+0000", 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.56923585200000004, 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01028242, 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 726 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.426 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.774937+0000", 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.774937+0000", 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.774939+0000", 2024-09-16T04:02:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.03e-06 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.775240+0000", 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030115699999999998 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.775264+0000", 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3828999999999998e-05 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.775274+0000", 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9760000000000007e-06 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.775384+0000", 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110326 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.775532+0000", 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014746099999999999 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.777824+0000", 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022918959999999999 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.779909+0000", 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020851060000000002 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.785108+0000", 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051994609999999998 2024-09-16T04:02:02.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.785219+0000", 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111178 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:731 3.ds0 3:bab6d723:::smithi18831615-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.786915+0000", 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.55725785100000003, 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0090938930000000005, 2024-09-16T04:02:02.431 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 731 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.786915+0000", 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.786915+0000", 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.786916+0000", 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6400000000000001e-07 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.786920+0000", 2024-09-16T04:02:02.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3429999999999998e-06 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.786931+0000", 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1280999999999999e-05 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.786939+0000", 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.525e-06 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.791298+0000", 2024-09-16T04:02:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043594649999999999 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.791404+0000", 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106124 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.794342+0000", 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002937664 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.794424+0000", 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1907e-05 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.796009+0000", 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00158482 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:761 3.ds0 3:b4418190:::smithi18831615-35:head [copy-from ver 27] snapc 1e=[1e,1d,1c,1b,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.942736+0000", 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.40143643400000001, 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028665875, 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 761 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942736+0000", 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942736+0000", 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942739+0000", 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2900000000000001e-06 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942749+0000", 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0920999999999999e-05 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942776+0000", 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6097000000000001e-05 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942787+0000", 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1676e-05 2024-09-16T04:02:02.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942929+0000", 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014148899999999999 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.943063+0000", 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000134454 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.962776+0000", 2024-09-16T04:02:02.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019712398999999999 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.967865+0000", 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050889569999999999 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.971298+0000", 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00343298 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.971402+0000", 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010461199999999999 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:762 3.ds0 3:b4418190:::smithi18831615-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.942824+0000", 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.40134859000000001, 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02255515, 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.442 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 762 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942824+0000", 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942824+0000", 2024-09-16T04:02:02.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942825+0000", 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-06 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942830+0000", 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9200000000000003e-06 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942848+0000", 2024-09-16T04:02:02.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7453e-05 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942866+0000", 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8603000000000001e-05 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.943371+0000", 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050457099999999999 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.943466+0000", 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5042000000000002e-05 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.965096+0000", 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021629808 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.965233+0000", 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013705400000000001 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.965379+0000", 2024-09-16T04:02:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014663299999999999 2024-09-16T04:02:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.447 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:02.448 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:02:02.450 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:02.445+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:02.549 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:02.545+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:02.576 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:02.577 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:02.577 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:02:02.577 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:02.590 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:02:02.651 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:02.645+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:02.733 INFO:tasks.rados.rados.0.smithi188.stdout:295: done (2 left) 2024-09-16T04:02:02.733 INFO:tasks.rados.rados.0.smithi188.stdout:296: rmattr oid 40 current snap is 29 2024-09-16T04:02:02.735 INFO:tasks.rados.rados.0.smithi188.stdout:294: expect (ObjNum 97 snap 19 seq_num 97) 2024-09-16T04:02:02.756 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:02.748+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:02.777 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:02:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:02:02.778 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:02.797 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:02:02.831 INFO:tasks.rados.rados.0.smithi188.stdout:293: finishing copy_from racing read to smithi18831615-35 2024-09-16T04:02:02.831 INFO:tasks.rados.rados.0.smithi188.stdout:293: finishing copy_from to smithi18831615-35 2024-09-16T04:02:02.831 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 27 (ObjNum 98 snap 21 seq_num 98) dirty exists 2024-09-16T04:02:02.832 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 40 v 85 (ObjNum 65 snap 0 seq_num 1600348519) dirty exists 2024-09-16T04:02:02.832 INFO:tasks.rados.rados.0.smithi188.stdout:293: done (2 left) 2024-09-16T04:02:02.832 INFO:tasks.rados.rados.0.smithi188.stdout:294: done (1 left) 2024-09-16T04:02:02.832 INFO:tasks.rados.rados.0.smithi188.stdout:296: done (0 left) 2024-09-16T04:02:02.832 INFO:tasks.rados.rados.0.smithi188.stdout:297: snap_create 2024-09-16T04:02:02.852 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:02.849+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:301 3.cs0 3:3922c872:::smithi18831615-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.001762378999999, 2024-09-16T04:02:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024704912999999998, 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 301 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911958+0000", 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911962+0000", 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.63e-06 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911976+0000", 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4027e-05 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911985+0000", 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7530000000000007e-06 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928563+0000", 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016578809 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928680+0000", 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117123 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931348+0000", 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002667141 2024-09-16T04:02:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931647+0000", 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029889700000000001 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.936662+0000", 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050156630000000001 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:365 3.cs0 3:389a65d8:::smithi18831615-2:head [read 0~32768,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.718790+0000", 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.194929412, 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021834770999999999, 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 365 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718790+0000", 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718790+0000", 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718794+0000", 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0729999999999998e-06 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718810+0000", 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5532e-05 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718844+0000", 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3949999999999999e-05 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718901+0000", 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7669e-05 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719047+0000", 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000145694 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719060+0000", 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.725018+0000", 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059579450000000001 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.725265+0000", 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024645500000000002 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.740625+0000", 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015360303 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:366 3.cs0 3:389a65d8:::smithi18831615-2:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.194770982000001, 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022413889999999999, 2024-09-16T04:02:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 366 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718950+0000", 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.663e-06 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718960+0000", 2024-09-16T04:02:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.893e-06 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718982+0000", 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1820000000000001e-05 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718999+0000", 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7280000000000001e-05 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719156+0000", 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015617699999999999 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719166+0000", 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0832999999999999e-05 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.730920+0000", 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01175319 2024-09-16T04:02:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.731179+0000", 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000259588 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.741363+0000", 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010183446000000001 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:367 3.cs0 3:389a65d8:::smithi18831615-2:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.194665571000002, 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024825495, 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 367 2024-09-16T04:02:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719056+0000", 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4839999999999999e-06 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719063+0000", 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7959999999999996e-06 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719085+0000", 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1365999999999998e-05 2024-09-16T04:02:02.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719100+0000", 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4863e-05 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719209+0000", 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108978 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719218+0000", 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1339999999999993e-06 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733259+0000", 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014041420000000001 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733873+0000", 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061368800000000004 2024-09-16T04:02:02.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.743880+0000", 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010006766 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:370 3.fs0 3:fec5116f:::smithi18831615-8:head [read 0~131072,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.719304+0000", 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.194415156000002, 2024-09-16T04:02:02.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022086333, 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 370 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719304+0000", 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719304+0000", 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719306+0000", 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.314e-06 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719313+0000", 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.396e-06 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719333+0000", 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9817e-05 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719349+0000", 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6429000000000001e-05 2024-09-16T04:02:02.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719415+0000", 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5778999999999995e-05 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719423+0000", 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5970000000000003e-06 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.731971+0000", 2024-09-16T04:02:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012548020999999999 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.732207+0000", 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000236596 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.741391+0000", 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091833839999999993 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:393 3.cs0 3:35c140c5:::smithi18831615-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873305+0000", 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.040414289000001, 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022093642, 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 393 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.946 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873305+0000", 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873305+0000", 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873306+0000", 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1900000000000003e-07 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873308+0000", 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4080000000000001e-06 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873315+0000", 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6560000000000003e-06 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873319+0000", 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2939999999999999e-06 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.880138+0000", 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068183810000000001 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.880209+0000", 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0953999999999999e-05 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.882434+0000", 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022255539999999998 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.882561+0000", 2024-09-16T04:02:02.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012717100000000001 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895399+0000", 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012837705 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:396 3.cs0 3:3631f100:::smithi18831615-24:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:02:02.951 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.040356122999999, 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024842948, 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 396 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:02:02.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.01e-07 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873368+0000", 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.092e-06 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873374+0000", 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5949999999999998e-06 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873378+0000", 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0949999999999998e-06 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.886123+0000", 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012745645999999999 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.886245+0000", 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012146699999999999 2024-09-16T04:02:02.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.898206+0000", 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011961652 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:401 3.cs0 3:3ac353cc:::smithi18831615-37:head [rollback 7] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.040229852, 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02229078, 2024-09-16T04:02:02.956 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 401 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1799999999999995e-07 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873492+0000", 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.657e-06 2024-09-16T04:02:02.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873499+0000", 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.131e-06 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873504+0000", 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.305e-06 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888149+0000", 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014644292999999999 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888207+0000", 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8229000000000001e-05 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888580+0000", 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037271 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.891242+0000", 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026624280000000001 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895627+0000", 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043851180000000003 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895781+0000", 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015339100000000001 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.961 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:402 3.cs0 3:3ac353cc:::smithi18831615-37:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.040209708999999, 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022474063999999998, 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 402 2024-09-16T04:02:02.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0499999999999999e-07 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873531+0000", 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0361e-05 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873538+0000", 2024-09-16T04:02:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2219999999999996e-06 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873543+0000", 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8189999999999999e-06 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889058+0000", 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015515467 2024-09-16T04:02:02.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889129+0000", 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0946999999999997e-05 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889436+0000", 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030638799999999999 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.891244+0000", 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001808289 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895920+0000", 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046766380000000003 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895984+0000", 2024-09-16T04:02:02.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3528000000000006e-05 2024-09-16T04:02:02.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.968 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:475 3.cs0 3:3f3e33cc:::smithi18831615-38:head [setxattr _header (57) in=7b] snapc 12=[12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:46.742977+0000", 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.170742936, 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025134200999999998, 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.969 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 475 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742977+0000", 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742977+0000", 2024-09-16T04:02:02.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742978+0000", 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0759999999999999e-06 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742984+0000", 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7440000000000004e-06 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742995+0000", 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1686e-05 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743004+0000", 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3639999999999999e-06 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.757412+0000", 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014408746 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.757586+0000", 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017383200000000001 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.758544+0000", 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00095736499999999995 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.761119+0000", 2024-09-16T04:02:02.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025749779999999999 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.767922+0000", 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068035869999999998 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.768111+0000", 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018882299999999999 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:476 3.cs0 3:3f3e33cc:::smithi18831615-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:46.743016+0000", 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.170703443000001, 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028572347000000001, 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 476 2024-09-16T04:02:02.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743016+0000", 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743016+0000", 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743017+0000", 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7300000000000005e-07 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743021+0000", 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8099999999999999e-06 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743031+0000", 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0665999999999999e-05 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743041+0000", 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0006999999999999e-05 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.759305+0000", 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016263283999999999 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.759504+0000", 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019926000000000001 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.768977+0000", 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0094726040000000008 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.769120+0000", 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000143022 2024-09-16T04:02:02.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.771589+0000", 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024689209999999998 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:514 3.cs0 3:35c140c5:::smithi18831615-48:head [read 0~2211840,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861354+0000", 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.052365811, 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022129296999999999, 2024-09-16T04:02:02.981 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 514 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861354+0000", 2024-09-16T04:02:02.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861354+0000", 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861358+0000", 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4730000000000002e-06 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861374+0000", 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5818999999999999e-05 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861425+0000", 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1047000000000001e-05 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861451+0000", 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5758999999999999e-05 2024-09-16T04:02:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861658+0000", 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020674800000000001 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861678+0000", 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0522000000000001e-05 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867064+0000", 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053854799999999998 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867293+0000", 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022948700000000001 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.883483+0000", 2024-09-16T04:02:02.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016189961999999999 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:515 3.cs0 3:35c140c5:::smithi18831615-48:head [read 0~2211840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861550+0000", 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.052169988999999, 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025758455, 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.987 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 515 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861550+0000", 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861550+0000", 2024-09-16T04:02:02.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861551+0000", 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.61e-06 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861562+0000", 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0633e-05 2024-09-16T04:02:02.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861602+0000", 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9660999999999999e-05 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861643+0000", 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1440999999999998e-05 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861797+0000", 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000153688 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861814+0000", 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7269e-05 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.868678+0000", 2024-09-16T04:02:02.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068639139999999996 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.868904+0000", 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000225644 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.887308+0000", 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018404594999999999 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.992 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:516 3.cs0 3:35c140c5:::smithi18831615-48:head [read 0~2211840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861724+0000", 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.051995968, 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030702040999999999, 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.993 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 516 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861724+0000", 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861724+0000", 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861725+0000", 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6360000000000001e-06 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861735+0000", 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2229999999999996e-06 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861757+0000", 2024-09-16T04:02:02.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2069000000000002e-05 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861774+0000", 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7379000000000001e-05 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861880+0000", 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010627 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:02.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861888+0000", 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2199999999999992e-06 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.869811+0000", 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007922208 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.870456+0000", 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00064530399999999997 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.892426+0000", 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021969731999999999 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:540 3.fs0 3:ff708987:::smithi18831615-20:head [copy-from ver 13] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.068918+0000", 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.844802080999999, 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034648133999999997, 2024-09-16T04:02:02.998 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 540 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068918+0000", 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:02.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068918+0000", 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068920+0000", 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.717e-06 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068934+0000", 2024-09-16T04:02:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3472000000000001e-05 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068966+0000", 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2085000000000002e-05 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068981+0000", 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069050+0000", 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8392000000000005e-05 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069121+0000", 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1513000000000005e-05 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.094700+0000", 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025578430999999999 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.101389+0000", 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066896339999999999 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.103448+0000", 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020585209999999998 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.103566+0000", 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117923 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:541 3.fs0 3:ff708987:::smithi18831615-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:03.004 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.069017+0000", 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.844702789999999, 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032194999000000002, 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 541 2024-09-16T04:02:03.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069017+0000", 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069017+0000", 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069018+0000", 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0130000000000001e-06 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069023+0000", 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8369999999999996e-06 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069036+0000", 2024-09-16T04:02:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2872e-05 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069045+0000", 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9400000000000008e-06 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069236+0000", 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000191101 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069260+0000", 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3986e-05 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.100647+0000", 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031387198999999998 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.100938+0000", 2024-09-16T04:02:03.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000290933 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.101212+0000", 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027411800000000001 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:649 3.cs0 3:30f8ffd7:::smithi18831615-11:head [copy-from ver 51] snapc 1b=[1b,1a,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:03.010 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.000111+0000", 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.913608967, 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.047605886, 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 649 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000111+0000", 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000111+0000", 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000112+0000", 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8000000000000004e-07 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000118+0000", 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5200000000000003e-06 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000128+0000", 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0302000000000001e-05 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000136+0000", 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5970000000000003e-06 2024-09-16T04:02:03.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.007587+0000", 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074513799999999996 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.007721+0000", 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013325300000000001 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.042660+0000", 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034939323000000001 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.046875+0000", 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042154849999999997 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.046987+0000", 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011165100000000001 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.047717+0000", 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072949500000000003 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:650 3.cs0 3:30f8ffd7:::smithi18831615-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.000146+0000", 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.9135740820000002, 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.044688439000000003, 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.016 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 650 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000146+0000", 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000146+0000", 2024-09-16T04:02:03.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000146+0000", 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000152+0000", 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2630000000000003e-06 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000161+0000", 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6600000000000007e-06 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000169+0000", 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8790000000000003e-06 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.007999+0000", 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078302379999999998 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.008035+0000", 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5336000000000002e-05 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.044510+0000", 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.036475664999999997 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.044663+0000", 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000152793 2024-09-16T04:02:03.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.044834+0000", 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017088499999999999 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:706 3.cs0 3:30f8ffd7:::smithi18831615-11:head [copy-from ver 71] snapc 1d=[1d,1c,1b,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.000875+0000", 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.912844191, 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.051306851000000001, 2024-09-16T04:02:03.021 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 706 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000875+0000", 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000875+0000", 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000879+0000", 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7100000000000001e-06 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000901+0000", 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1937999999999999e-05 2024-09-16T04:02:03.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000938+0000", 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6600999999999997e-05 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000956+0000", 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8437e-05 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001117+0000", 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016098499999999999 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001131+0000", 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3597e-05 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.004495+0000", 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033642619999999998 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.004767+0000", 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000271625 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.037080+0000", 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032313596 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.045271+0000", 2024-09-16T04:02:03.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081910279999999995 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.052078+0000", 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068070939999999996 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.052182+0000", 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103978 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:707 3.cs0 3:30f8ffd7:::smithi18831615-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.001018+0000", 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.9127018279999999, 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.043917128999999999, 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.028 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 707 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001018+0000", 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001018+0000", 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001019+0000", 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.029e-06 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001028+0000", 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9069999999999996e-06 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001047+0000", 2024-09-16T04:02:03.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8890999999999999e-05 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001058+0000", 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1242e-05 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001228+0000", 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017024499999999999 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001237+0000", 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1160000000000005e-06 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.005191+0000", 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039534160000000004 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.005350+0000", 2024-09-16T04:02:03.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000159273 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.044337+0000", 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.038987496000000003 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.044624+0000", 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028676400000000001 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.044935+0000", 2024-09-16T04:02:03.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031074999999999999 2024-09-16T04:02:03.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.034 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.034 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.034 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:03.035 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:02:03.037 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:02:03.037 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:03.033+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:03.138 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:03.133+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:03.195 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:03.195 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:03.205 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:03.205 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:03.205 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:02:03.205 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:03.215 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:03.216 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:03.219 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:02:03.236 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:03.237 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:03.238 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:03.237+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:03.254 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:03.254 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:03.274 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:03.274 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:03.296 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:03.297 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:03.318 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:03.319 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:03.339 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:03.337+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:03.340 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:03.340 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:03.342 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:03.342 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:03.342 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:02:03.343 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:02:03.343 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:03.357 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:02:03.362 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:03.362 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:03.385 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:03.386 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:03.443 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:03.441+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:03.527 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:108 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.589947975999998, 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014725059, 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 108 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2399999999999998e-07 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933259+0000", 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.937e-06 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933267+0000", 2024-09-16T04:02:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6229999999999997e-06 2024-09-16T04:02:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933273+0000", 2024-09-16T04:02:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-09-16T04:02:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936240+0000", 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002967365 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936330+0000", 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0200999999999999e-05 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.940982+0000", 2024-09-16T04:02:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046523210000000001 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.941058+0000", 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5798999999999997e-05 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.947981+0000", 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006922615 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:141 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.542156724999998, 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087077830000000002, 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 141 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981047+0000", 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981048+0000", 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:02:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981051+0000", 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1659999999999998e-06 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981057+0000", 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6229999999999999e-06 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.981063+0000", 2024-09-16T04:02:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6550000000000001e-06 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.983220+0000", 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021567890000000001 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.983291+0000", 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1326000000000001e-05 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987840+0000", 2024-09-16T04:02:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045492309999999999 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.987922+0000", 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1205000000000006e-05 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.989755+0000", 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018333049999999999 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:172 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.494733779999997, 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091168310000000006, 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 172 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028470+0000", 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9900000000000001e-07 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028474+0000", 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6969999999999999e-06 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028482+0000", 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6420000000000007e-06 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.028490+0000", 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.712e-06 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030305+0000", 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001815327 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.030412+0000", 2024-09-16T04:02:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107055 2024-09-16T04:02:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036597+0000", 2024-09-16T04:02:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061849360000000003 2024-09-16T04:02:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.036681+0000", 2024-09-16T04:02:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4110999999999995e-05 2024-09-16T04:02:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:22.037587+0000", 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090585199999999996 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:314 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~491520] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.512556611000001, 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013929950999999999, 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 314 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010651+0000", 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.467e-06 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011198+0000", 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054779799999999995 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011237+0000", 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8092000000000001e-05 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011254+0000", 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7232999999999999e-05 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011454+0000", 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020057300000000001 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011623+0000", 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016875899999999999 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014078+0000", 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024548510000000001 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024181+0000", 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010102877 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024366+0000", 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018482199999999999 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024577+0000", 2024-09-16T04:02:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021147900000000001 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:315 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~770048] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.506958754999999, 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091517409999999997, 2024-09-16T04:02:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 315 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016245+0000", 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.016247+0000", 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1799999999999999e-06 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017297+0000", 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010498269999999999 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017327+0000", 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0133000000000001e-05 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017349+0000", 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017547+0000", 2024-09-16T04:02:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019868199999999999 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017839+0000", 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029153000000000002 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.020248+0000", 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024092649999999998 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024592+0000", 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043436849999999999 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025288+0000", 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069595499999999995 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025397+0000", 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010890799999999999 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:316 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (56) in=7b] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.505790916999999, 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010644074999999999, 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 316 2024-09-16T04:02:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017415+0000", 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.99e-06 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017424+0000", 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4129999999999995e-06 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017444+0000", 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0115999999999998e-05 2024-09-16T04:02:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017463+0000", 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8199999999999999e-05 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022150+0000", 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046871079999999997 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022414+0000", 2024-09-16T04:02:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000264491 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023228+0000", 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081356500000000003 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025417+0000", 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021892959999999999 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.027816+0000", 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023989509999999999 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028057+0000", 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000240945 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:317 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.505679843000003, 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017770083999999998, 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 317 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017526+0000", 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017533+0000", 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1960000000000002e-06 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017555+0000", 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2157000000000001e-05 2024-09-16T04:02:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017574+0000", 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8298999999999999e-05 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024692+0000", 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071183920000000003 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024964+0000", 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027219700000000001 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029509+0000", 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045448850000000002 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031984+0000", 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024747929999999999 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.035294+0000", 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033101850000000002 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s4 51/33 ECSubWrite(tid=75, reqid=client.4646.0:384, at_version=51'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.852557+0000", 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.670646454, 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087782580000000006, 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852557+0000", 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852557+0000", 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852560+0000", 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8200000000000001e-06 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853239+0000", 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067888699999999998 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853289+0000", 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0346000000000002e-05 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853317+0000", 2024-09-16T04:02:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7203e-05 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853503+0000", 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018615600000000001 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853530+0000", 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6896000000000001e-05 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.861245+0000", 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077158360000000002 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.861336+0000", 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0113999999999998e-05 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:510 3.5s0 3:a51a87ac:::smithi18831615-36:head [copy-from ver 36] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861100+0000", 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.662103397999999, 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.041038472999999999, 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 510 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861100+0000", 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861100+0000", 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861104+0000", 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3799999999999998e-06 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861121+0000", 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7459e-05 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861172+0000", 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.126e-05 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861200+0000", 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7138999999999999e-05 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861474+0000", 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027421100000000001 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861498+0000", 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4468999999999999e-05 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.866976+0000", 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054775709999999997 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867232+0000", 2024-09-16T04:02:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025570900000000001 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.895930+0000", 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028698860999999999 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.900460+0000", 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004529178 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.901938+0000", 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014787699999999999 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.902139+0000", 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000200466 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:511 3.5s0 3:a51a87ac:::smithi18831615-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861259+0000", 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.661944573, 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.038909821999999997, 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 511 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861259+0000", 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861259+0000", 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861261+0000", 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6029999999999999e-06 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861269+0000", 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8420000000000005e-06 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861303+0000", 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4662000000000001e-05 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861322+0000", 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902e-05 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861670+0000", 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034817399999999998 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861687+0000", 2024-09-16T04:02:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6626e-05 2024-09-16T04:02:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867660+0000", 2024-09-16T04:02:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059733540000000002 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867771+0000", 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011105 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.899915+0000", 2024-09-16T04:02:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032144104 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.900038+0000", 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000122679 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.900169+0000", 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013082600000000001 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s3 66/33 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=66'32, trim_to=0'0, roll_forward_to=63'30))", 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.737038+0000", 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.786165330999999, 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011633279999999999, 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737038+0000", 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737038+0000", 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737039+0000", 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:02:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737092+0000", 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3233999999999999e-05 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737117+0000", 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4782000000000001e-05 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737140+0000", 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2898e-05 2024-09-16T04:02:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737324+0000", 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018401999999999999 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737349+0000", 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5021999999999999e-05 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.748605+0000", 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011255644 2024-09-16T04:02:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.748672+0000", 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7096999999999998e-05 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:563 3.5s0 3:a37b04c5:::smithi18831615-47:head [append 0~0] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:52.761739+0000", 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.761465024, 2024-09-16T04:02:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010909632000000001, 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 563 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761739+0000", 2024-09-16T04:02:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761739+0000", 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761742+0000", 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9579999999999999e-06 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761752+0000", 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0662999999999999e-05 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761775+0000", 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2293000000000002e-05 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761788+0000", 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3074000000000001e-05 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761939+0000", 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015136500000000001 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.762175+0000", 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023601300000000001 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.763620+0000", 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001444877 2024-09-16T04:02:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.768564+0000", 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049439870000000004 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.772454+0000", 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003889783 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.772648+0000", 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019461899999999999 2024-09-16T04:02:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:564 3.5s0 3:a37b04c5:::smithi18831615-47:head [rollback e] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:52.761815+0000", 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.761388425, 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015990183000000002, 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 564 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761815+0000", 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761815+0000", 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761816+0000", 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6499999999999998e-07 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761821+0000", 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761834+0000", 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3643e-05 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761842+0000", 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1130000000000001e-06 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.765701+0000", 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038581029999999999 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.765961+0000", 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026021099999999999 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767423+0000", 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001461979 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.772787+0000", 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053647039999999997 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.777603+0000", 2024-09-16T04:02:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048160119999999997 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.777806+0000", 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020206900000000001 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:565 3.5s0 3:a37b04c5:::smithi18831615-47:head [append 0~0] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:52.761896+0000", 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.761307452, 2024-09-16T04:02:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017049812000000001, 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 565 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761896+0000", 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761896+0000", 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761897+0000", 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1399999999999996e-07 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761902+0000", 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4190000000000001e-06 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761915+0000", 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2998e-05 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761924+0000", 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1750000000000005e-06 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.769117+0000", 2024-09-16T04:02:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071931879999999997 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.769446+0000", 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032917200000000003 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.770752+0000", 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013064000000000001 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.773481+0000", 2024-09-16T04:02:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027282980000000001 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.778765+0000", 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052845219999999998 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.778946+0000", 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018092600000000001 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:618 3.5s0 3:a51a87ac:::smithi18831615-36:head [copy-from ver 31] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928147+0000", 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.5950569760000004, 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015153856, 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 618 2024-09-16T04:02:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928147+0000", 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928147+0000", 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928151+0000", 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8299999999999998e-06 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928167+0000", 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6152999999999999e-05 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928203+0000", 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6554000000000003e-05 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928226+0000", 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3142999999999998e-05 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928475+0000", 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024883300000000001 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928758+0000", 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000282883 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.938731+0000", 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099730040000000006 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.942778+0000", 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040466529999999999 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.943074+0000", 2024-09-16T04:02:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029642900000000002 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.943301+0000", 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000226374 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:619 3.5s0 3:a51a87ac:::smithi18831615-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928283+0000", 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.5949204689999998, 2024-09-16T04:02:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013685522, 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 619 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928283+0000", 2024-09-16T04:02:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928283+0000", 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928285+0000", 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.623e-06 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928292+0000", 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2400000000000001e-06 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928316+0000", 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3462e-05 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928335+0000", 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9575000000000001e-05 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.929213+0000", 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00087792499999999999 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.929415+0000", 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020199000000000001 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941419+0000", 2024-09-16T04:02:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012004182 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941669+0000", 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000249924 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941969+0000", 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029960099999999998 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:632 3.5s0 3:a37b04c5:::smithi18831615-47:head [create,setxattr [OXS[Rf]kSmSVZkcqrPoTRUsq[o]epTblNRaQggt (1858) in=40b,setxattr QrooaVmb\\N]q[ho]kgWWcr\\eocaQRtZ_`]Qsipqt (174) in=40b,setxattr Ydefbt^YQ\\dpmjectRYX^qRe[eQdQqbZhllc (770) in=36b,setxattr TV\\gUfWjiSV[PabStYoPOXcXUVsSaXdhhRnnhpfs (454) in=40b,setxattr m_O\\\\icSfcPOnoNRZ]O`X]X`ZaVOXegXNVmPglmp (734) in=40b,setxattr mf`_l`a[Y`agdjhtOpoWmnWT^dS[ScSdZWamrcjp (699) in=40b,setxattr hPm]lVO[\\[iefNWiNnaWNQr (770) in=23b,setxattr gkna_]qPX[S`[_g_ifYo]hUlgScp_jnl[^WpdheP (1614) in=40b,setxattr YSndfaQqlVQQXmZiNjPrZZa`glYedsbkkcroSttS (1333) in=40b,setxattr VVliPgssabe (770) in=11b,setxattr kdelpR\\kNn^\\r\\aeabqa\\jcQSaOXgNlVTroTaodb (1610) in=40b,setxattr a[YZZ[_nQae][Ujatlhnen_QorP_PROiQkaafPl^ (1754) in=40b,setxattr Y\\iYcXa`paqfZggTnSmaVZ[Yt`te\\rdepg^foUUX (12280) in=40b,setxattr WPicPk_V\\Z]`USR]\\Xq (770) in=19b,setxattr gqtnZjUUjhkmnQqhohYVhkliPOppPSXTshf_sVbq (877) in=40b,setxattr Nadgio[acb^P_fdps]bigQrctbcktsZdTe]Z_hhb (8628) in=40b,setxattr oUUbUt^qtWedq]`NoRbZf[jPhdNrRbjTU`m^jbpe (666) in=40b,setxattr UUhYqmbd_S[all[PkPg\\odhP\\gkshjetOj][^OWq (16140) in=40b,setxattr WSTftae_cS`q^sZiZQskrTnpRX^Tkmid^ZbYOeTo (1997) in=40b,setxattr bdcqYYhcTQq^O[TjlUb[oTXa_hoaQStVglem^VqX (981) in=40b] snapc 1a=[1a,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:56.940054+0000", 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.5831498340000003, 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015103711000000001, 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 632 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940054+0000", 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940054+0000", 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940058+0000", 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9960000000000004e-06 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940158+0000", 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9825000000000001e-05 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940209+0000", 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0788000000000002e-05 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940235+0000", 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6137999999999999e-05 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940406+0000", 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017153000000000001 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940423+0000", 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7241999999999999e-05 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.944927+0000", 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045031070000000001 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.945295+0000", 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036849799999999998 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.948503+0000", 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003208397 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.954274+0000", 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057702350000000003 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.954431+0000", 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015692500000000001 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.955158+0000", 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072703000000000004 2024-09-16T04:02:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:665 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.148626+0000", 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.3745776080000001, 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014036061000000001, 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 665 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148626+0000", 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148626+0000", 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148630+0000", 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4139999999999998e-06 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148647+0000", 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7419e-05 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148694+0000", 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7209999999999997e-05 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148717+0000", 2024-09-16T04:02:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2364000000000001e-05 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148833+0000", 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000116414 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148974+0000", 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014129100000000001 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.162662+0000", 2024-09-16T04:02:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013687949 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:666 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.148763+0000", 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.3744406600000003, 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0150174, 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 666 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148763+0000", 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148763+0000", 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148765+0000", 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.466e-06 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148773+0000", 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3620000000000006e-06 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148792+0000", 2024-09-16T04:02:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8799000000000001e-05 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148807+0000", 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5325e-05 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.156052+0000", 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072448499999999997 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.156280+0000", 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000228536 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.163780+0000", 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007500062 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:667 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.148837+0000", 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.3743669870000002, 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015477739000000001, 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 667 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148837+0000", 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148837+0000", 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148838+0000", 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.012e-06 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148847+0000", 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7309999999999999e-06 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148864+0000", 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7748e-05 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148878+0000", 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3848e-05 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.157464+0000", 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085862760000000003 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.157707+0000", 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000242215 2024-09-16T04:02:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.164315+0000", 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066079090000000004 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:03.640 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:03.641 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:03.643 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:02:03.637+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:02:03.643 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:02:03.649 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:03.649 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:03.685 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:03.685 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:03.719 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:03.719 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:03.743 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:03.741+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:03.744 INFO:tasks.rados.rados.0.smithi188.stdout:297: done (0 left) 2024-09-16T04:02:03.745 INFO:tasks.rados.rados.0.smithi188.stdout:298: rollback oid 7 current snap is 30 2024-09-16T04:02:03.745 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 7 to 5 2024-09-16T04:02:03.745 INFO:tasks.rados.rados.0.smithi188.stdout:299: snap_create 2024-09-16T04:02:03.752 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:02:03.754 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:03.755 INFO:tasks.rados.rados.0.smithi188.stdout:298: finishing rollback tid 0 to smithi18831615-7 2024-09-16T04:02:03.755 INFO:tasks.rados.rados.0.smithi188.stdout:298: finishing rollback tid 1 to smithi18831615-7 2024-09-16T04:02:03.756 INFO:tasks.rados.rados.0.smithi188.stdout:298: finishing rollback tid 2 to smithi18831615-7 2024-09-16T04:02:03.756 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 7 v 63 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T04:02:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:02:03.752+0000 7f91ee91d640 0 -- 172.21.15.37:0/3709276061 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7f91e8226410 msgr2=0x7f91e8228800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T04:02:03.792 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:03.793 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:03.808 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:02:03.844 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:02:03.841+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:02:03.945 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:03.941+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:03.988 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:03.989 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:03.989 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:03.990 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:04.005 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:02:04.057 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:04.056+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:04.136 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:04.137 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:04.147 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:04.144+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:04.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:02:04.247 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:04.241+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:04.323 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:04.323 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:04.337 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:02:04.348 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:02:04.345+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:02:04.449 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:04.445+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:04.466 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:04.466 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:04.480 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:02:04.550 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:04.548+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:04.607 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:04.607 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:04.621 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:02:04.650 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:04.645+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:04.737 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-09-16T04:02:04.743 INFO:tasks.rados.rados.0.smithi188.stdout:298: done (1 left) 2024-09-16T04:02:04.743 INFO:tasks.rados.rados.0.smithi188.stdout:299: done (0 left) 2024-09-16T04:02:04.743 INFO:tasks.rados.rados.0.smithi188.stdout:300: rollback oid 3 current snap is 31 2024-09-16T04:02:04.743 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 3 to 20 2024-09-16T04:02:04.743 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 19 current snap is 31 2024-09-16T04:02:04.743 INFO:tasks.rados.rados.0.smithi188.stdout:301: seq_num 125 ranges {1654784=557056,2211840=475136,2686976=507904,3194880=475136,3670016=98304} 2024-09-16T04:02:04.751 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:04.748+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:04.776 INFO:tasks.rados.rados.0.smithi188.stdout:301: writing smithi18831615-19 from 1654784 to 2211840 tid 1 2024-09-16T04:02:04.783 INFO:tasks.rados.rados.0.smithi188.stdout:301: writing smithi18831615-19 from 2211840 to 2686976 tid 2 2024-09-16T04:02:04.792 INFO:tasks.rados.rados.0.smithi188.stdout:301: writing smithi18831615-19 from 2686976 to 3194880 tid 3 2024-09-16T04:02:04.796 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:04.796 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:04.799 INFO:tasks.rados.rados.0.smithi188.stdout:301: writing smithi18831615-19 from 3194880 to 3670016 tid 4 2024-09-16T04:02:04.800 INFO:tasks.rados.rados.0.smithi188.stdout:301: writing smithi18831615-19 from 3670016 to 3768320 tid 5 2024-09-16T04:02:04.801 INFO:tasks.rados.rados.0.smithi188.stdout:302: copy_from oid 24 from oid 10 current snap is 31 2024-09-16T04:02:04.801 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 30 current snap is 31 2024-09-16T04:02:04.801 INFO:tasks.rados.rados.0.smithi188.stdout:303: seq_num 126 ranges {491520=573440,1064960=622592,1687552=622592,2310144=311296} 2024-09-16T04:02:04.811 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:02:04.812 INFO:tasks.rados.rados.0.smithi188.stdout:303: writing smithi18831615-30 from 491520 to 1064960 tid 1 2024-09-16T04:02:04.819 INFO:tasks.rados.rados.0.smithi188.stdout:303: writing smithi18831615-30 from 1064960 to 1687552 tid 2 2024-09-16T04:02:04.824 INFO:tasks.rados.rados.0.smithi188.stdout:303: writing smithi18831615-30 from 1687552 to 2310144 tid 3 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:303: writing smithi18831615-30 from 2310144 to 2621440 tid 4 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:304: rollback oid 27 current snap is 31 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 27 to 17 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:300: finishing rollback tid 0 to smithi18831615-3 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:300: finishing rollback tid 1 to smithi18831615-3 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 3 v 94 (ObjNum 49 snap 0 seq_num 0) dirty dne 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:301: finishing write tid 1 to smithi18831615-19 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:301: finishing write tid 2 to smithi18831615-19 2024-09-16T04:02:04.828 INFO:tasks.rados.rados.0.smithi188.stdout:301: finishing write tid 3 to smithi18831615-19 2024-09-16T04:02:04.829 INFO:tasks.rados.rados.0.smithi188.stdout:301: finishing write tid 4 to smithi18831615-19 2024-09-16T04:02:04.829 INFO:tasks.rados.rados.0.smithi188.stdout:301: finishing write tid 5 to smithi18831615-19 2024-09-16T04:02:04.829 INFO:tasks.rados.rados.0.smithi188.stdout:300: done (4 left) 2024-09-16T04:02:04.829 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 11 current snap is 31 2024-09-16T04:02:04.829 INFO:tasks.rados.rados.0.smithi188.stdout:305: seq_num 127 ranges {3358720=589824,3948544=753664,4702208=475136,5177344=327680} 2024-09-16T04:02:04.852 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:04.849+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:04.862 INFO:tasks.rados.rados.0.smithi188.stdout:305: writing smithi18831615-11 from 3358720 to 3948544 tid 1 2024-09-16T04:02:04.869 INFO:tasks.rados.rados.0.smithi188.stdout:305: writing smithi18831615-11 from 3948544 to 4702208 tid 2 2024-09-16T04:02:04.873 INFO:tasks.rados.rados.0.smithi188.stdout:305: writing smithi18831615-11 from 4702208 to 5177344 tid 3 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:305: writing smithi18831615-11 from 5177344 to 5505024 tid 4 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:306: delete oid 42 current snap is 31 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:301: finishing write tid 6 to smithi18831615-19 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:302: finishing copy_from racing read to smithi18831615-24 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:301: finishing write tid 7 to smithi18831615-19 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:301: oid 19 updating version 0 to 32 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:301: oid 19 version 32 is already newer than 29 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:301: oid 19 updating version 32 to 33 2024-09-16T04:02:04.876 INFO:tasks.rados.rados.0.smithi188.stdout:301: oid 19 version 33 is already newer than 30 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:301: oid 19 updating version 33 to 34 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:301: oid 19 version 34 is already newer than 31 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 19 v 34 (ObjNum 125 snap 31 seq_num 125) dirty exists 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:301: left oid 19 (ObjNum 125 snap 31 seq_num 125) 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:302: finishing copy_from to smithi18831615-24 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 24 v 31 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:303: finishing write tid 1 to smithi18831615-30 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:303: finishing write tid 2 to smithi18831615-30 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:303: finishing write tid 3 to smithi18831615-30 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:303: finishing write tid 4 to smithi18831615-30 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:303: finishing write tid 5 to smithi18831615-30 2024-09-16T04:02:04.877 INFO:tasks.rados.rados.0.smithi188.stdout:303: finishing write tid 6 to smithi18831615-30 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:303: oid 30 updating version 0 to 30 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:303: oid 30 version 30 is already newer than 28 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:303: oid 30 updating version 30 to 31 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:303: oid 30 version 31 is already newer than 29 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:303: oid 30 version 31 is already newer than 27 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 30 v 31 (ObjNum 126 snap 31 seq_num 126) dirty exists 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:303: left oid 30 (ObjNum 126 snap 31 seq_num 126) 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:304: finishing rollback tid 0 to smithi18831615-27 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:304: finishing rollback tid 1 to smithi18831615-27 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:304: finishing rollback tid 2 to smithi18831615-27 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 27 v 21 (ObjNum 90 snap 16 seq_num 90) dirty exists 2024-09-16T04:02:04.878 INFO:tasks.rados.rados.0.smithi188.stdout:305: finishing write tid 1 to smithi18831615-11 2024-09-16T04:02:04.883 INFO:tasks.rados.rados.0.smithi188.stdout:305: finishing write tid 2 to smithi18831615-11 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:301: done (5 left) 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:302: done (4 left) 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:303: done (3 left) 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:304: done (2 left) 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:306: done (1 left) 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:307: rollback oid 16 current snap is 31 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 16 to 12 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:308: rollback oid 35 current snap is 31 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 35 to 23 2024-09-16T04:02:04.884 INFO:tasks.rados.rados.0.smithi188.stdout:309: delete oid 15 current snap is 31 2024-09-16T04:02:04.885 INFO:tasks.rados.rados.0.smithi188.stdout:305: finishing write tid 3 to smithi18831615-11 2024-09-16T04:02:04.889 INFO:tasks.rados.rados.0.smithi188.stdout:305: finishing write tid 4 to smithi18831615-11 2024-09-16T04:02:04.889 INFO:tasks.rados.rados.0.smithi188.stdout:305: finishing write tid 5 to smithi18831615-11 2024-09-16T04:02:04.892 INFO:tasks.rados.rados.0.smithi188.stdout:309: done (3 left) 2024-09-16T04:02:04.892 INFO:tasks.rados.rados.0.smithi188.stdout:310: setattr oid 25 current snap is 31 2024-09-16T04:02:04.894 INFO:tasks.rados.rados.0.smithi188.stdout:311: snap_create 2024-09-16T04:02:04.894 INFO:tasks.rados.rados.0.smithi188.stdout:308: finishing rollback tid 0 to smithi18831615-35 2024-09-16T04:02:04.895 INFO:tasks.rados.rados.0.smithi188.stdout:307: finishing rollback tid 0 to smithi18831615-16 2024-09-16T04:02:04.895 INFO:tasks.rados.rados.0.smithi188.stdout:307: finishing rollback tid 1 to smithi18831615-16 2024-09-16T04:02:04.896 INFO:tasks.rados.rados.0.smithi188.stdout:305: finishing write tid 6 to smithi18831615-11 2024-09-16T04:02:04.896 INFO:tasks.rados.rados.0.smithi188.stdout:305: oid 11 updating version 0 to 205 2024-09-16T04:02:04.896 INFO:tasks.rados.rados.0.smithi188.stdout:305: oid 11 updating version 205 to 207 2024-09-16T04:02:04.896 INFO:tasks.rados.rados.0.smithi188.stdout:305: oid 11 version 207 is already newer than 206 2024-09-16T04:02:04.896 INFO:tasks.rados.rados.0.smithi188.stdout:305: oid 11 updating version 207 to 209 2024-09-16T04:02:04.897 INFO:tasks.rados.rados.0.smithi188.stdout:305: oid 11 version 209 is already newer than 208 2024-09-16T04:02:04.897 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 11 v 209 (ObjNum 127 snap 31 seq_num 127) dirty exists 2024-09-16T04:02:04.897 INFO:tasks.rados.rados.0.smithi188.stdout:305: left oid 11 (ObjNum 127 snap 31 seq_num 127) 2024-09-16T04:02:04.897 INFO:tasks.rados.rados.0.smithi188.stdout:308: finishing rollback tid 1 to smithi18831615-35 2024-09-16T04:02:04.897 INFO:tasks.rados.rados.0.smithi188.stdout:307: finishing rollback tid 2 to smithi18831615-16 2024-09-16T04:02:04.897 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 16 v 141 (ObjNum 80 snap 10 seq_num 80) dirty exists 2024-09-16T04:02:04.898 INFO:tasks.rados.rados.0.smithi188.stdout:308: finishing rollback tid 2 to smithi18831615-35 2024-09-16T04:02:04.898 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 67 (ObjNum 90 snap 16 seq_num 90) dirty exists 2024-09-16T04:02:04.902 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 25 v 28 (ObjNum 97 snap 19 seq_num 97) dirty exists 2024-09-16T04:02:04.953 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:04.949+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:04.988 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:04.988 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:05.002 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:02:05.053 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:05.052+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:05.127 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:05.128 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:05.142 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:02:05.186 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:05.184+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:05.256 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:05.252+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:05.315 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:05.315 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:05.330 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:02:05.356 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:05.353+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:05.457 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:05.456+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:05.500 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:05.500 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:05.514 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:02:05.558 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:05.557+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:05.659 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:05.656+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:05.679 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:05.679 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:05.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:02:05.760 INFO:tasks.rados.rados.0.smithi188.stdout:305: done (4 left) 2024-09-16T04:02:05.760 INFO:tasks.rados.rados.0.smithi188.stdout:307: done (3 left) 2024-09-16T04:02:05.760 INFO:tasks.rados.rados.0.smithi188.stdout:308: done (2 left) 2024-09-16T04:02:05.760 INFO:tasks.rados.rados.0.smithi188.stdout:310: done (1 left) 2024-09-16T04:02:05.761 INFO:tasks.rados.rados.0.smithi188.stdout:311: done (0 left) 2024-09-16T04:02:05.761 INFO:tasks.rados.rados.0.smithi188.stdout:312: setattr oid 29 current snap is 32 2024-09-16T04:02:05.761 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:05.757+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:05.762 INFO:tasks.rados.rados.0.smithi188.stdout:313: snap_remove snap 17 2024-09-16T04:02:05.833 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:05.833 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:05.848 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:02:05.861 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:05.856+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:05.962 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:02:05.962 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:02:05.962 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T04:02:05.963 INFO:tasks.thrashosds.thrasher:Setting osd 12 primary_affinity to 0.061646 2024-09-16T04:02:05.963 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 0.0616456720843922 2024-09-16T04:02:05.963 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:05.960+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:05.967 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:05.968 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:05.982 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:02:06.015 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:02:06.063 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:06.060+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:06.109 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:06.109 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:06.123 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:06.164 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:06.160+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:06.183 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:06.183 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:06.203 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:06.204 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:06.225 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:06.225 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:06.241 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:06.241 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:06.250 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:06.250 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:06.259 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:06.265 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:02:06.266 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:06.260+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:06.279 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:06.280 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:06.300 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:06.300 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:06.316 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:06.334 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:06.335 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:06.356 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:06.366 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:06.364+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:06.378 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:06.378 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:06.392 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:06.392 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:06.392 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:02:06.392 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:02:06.392 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:06.398 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:06.408 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:02:06.419 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:06.419 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:06.441 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:06.442 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:06.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:02:06.440+0000 7f3b3298a640 0 -- 172.21.15.37:0/41760783 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7f3b2c226180 msgr2=0x7f3b2c228570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:02:06.466 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:06.465+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:06.537 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:06.537 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:02:06.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:02:06.537 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:02:06.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.537 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4371.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.815860397000002, 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01704027, 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4371", 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.37:0/2503621667", 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716696+0000", 2024-09-16T04:02:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716711+0000", 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4676e-05 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716743+0000", 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1835000000000003e-05 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716769+0000", 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6327999999999999e-05 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716973+0000", 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020410900000000001 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.717006+0000", 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3084999999999999e-05 2024-09-16T04:02:06.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.724942+0000", 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0079355829999999995 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.724964+0000", 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1804999999999999e-05 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730286+0000", 2024-09-16T04:02:06.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053221689999999999 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730641+0000", 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035484999999999998 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730957+0000", 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0003165 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:06.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.732250+0000", 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001293039 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733535+0000", 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001284477 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733616+0000", 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0964999999999995e-05 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733727+0000", 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011134 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:458 3.8s0 3:1a33c5ec:::smithi18831615-27:head [append 0~753664] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.776616+0000", 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.755931200999999, 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022631235, 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.545 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 458 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.776616+0000", 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.776616+0000", 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.776619+0000", 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.456e-06 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.777851+0000", 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001231418 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.777889+0000", 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8513999999999998e-05 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.777909+0000", 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9806e-05 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.778069+0000", 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016032500000000001 2024-09-16T04:02:06.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.778084+0000", 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4606e-05 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782203+0000", 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041192720000000002 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782403+0000", 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000199643 2024-09-16T04:02:06.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.785240+0000", 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0028370359999999998 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.798715+0000", 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013474709 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.798949+0000", 2024-09-16T04:02:06.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023443800000000001 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.799247+0000", 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029801200000000001 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:459 3.8s0 3:1a33c5ec:::smithi18831615-27:head [append 0~720896] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.780773+0000", 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.751773754999999, 2024-09-16T04:02:06.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020809588, 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 459 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.780773+0000", 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.780773+0000", 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.780776+0000", 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3530000000000002e-06 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782262+0000", 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0014862779999999999 2024-09-16T04:02:06.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782301+0000", 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8977999999999998e-05 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782323+0000", 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2311999999999999e-05 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.788336+0000", 2024-09-16T04:02:06.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060125550000000002 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.788625+0000", 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00028906500000000003 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.791136+0000", 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002511503 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.799394+0000", 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082580050000000006 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.801480+0000", 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002086077 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.801583+0000", 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000102462 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:460 3.8s0 3:1a33c5ec:::smithi18831615-27:head [append 0~49152] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.782369+0000", 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.750178322, 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019757647, 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.557 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 460 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782369+0000", 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782369+0000", 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782371+0000", 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7829999999999999e-06 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782485+0000", 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000114314 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782510+0000", 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5445000000000001e-05 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782532+0000", 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1869e-05 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.794174+0000", 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011642199000000001 2024-09-16T04:02:06.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.794422+0000", 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024799400000000002 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.795963+0000", 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015403019999999999 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.799397+0000", 2024-09-16T04:02:06.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034340759999999999 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.801980+0000", 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025832889999999999 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.802126+0000", 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014637600000000001 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:461 3.8s0 3:1a33c5ec:::smithi18831615-27:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.782561+0000", 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.749986205999999, 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020044916999999999, 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 461 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782561+0000", 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782561+0000", 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782562+0000", 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4899999999999999e-06 2024-09-16T04:02:06.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782585+0000", 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2918999999999999e-05 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782605+0000", 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9502999999999999e-05 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782619+0000", 2024-09-16T04:02:06.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4459e-05 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.796957+0000", 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014337242 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.797155+0000", 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019887999999999999 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.798133+0000", 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0009780870000000001 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.799665+0000", 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001531632 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.802456+0000", 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027904620000000001 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.802606+0000", 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015024299999999999 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:462 3.8s0 3:1a33c5ec:::smithi18831615-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.782640+0000", 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.74990717, 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028545022, 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 462 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782640+0000", 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782640+0000", 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782641+0000", 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1570000000000001e-06 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782647+0000", 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3430000000000001e-06 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782668+0000", 2024-09-16T04:02:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0292000000000002e-05 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782683+0000", 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5362999999999998e-05 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.800227+0000", 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017543616000000001 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:06.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.800355+0000", 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012834000000000001 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.803488+0000", 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003132485 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.803618+0000", 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130514 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.811185+0000", 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0075669120000000003 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 61/33 ECSubWrite(tid=194, reqid=client.4646.0:472, at_version=61'118, trim_to=0'0, roll_forward_to=60'116))", 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.743931+0000", 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.788616585, 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0087012140000000005, 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.573 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.743931+0000", 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.743931+0000", 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.743931+0000", 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5600000000000001e-07 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.744661+0000", 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00073047800000000001 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.744687+0000", 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5911e-05 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.744703+0000", 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6189999999999999e-05 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.744881+0000", 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017717399999999999 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.744911+0000", 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0015e-05 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.752503+0000", 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0075923830000000003 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.752632+0000", 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012870700000000001 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.578 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(osd.11.22:1 3.8s0 3:1a33c5ec:::smithi18831615-27:head [assert-version v13,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.069880+0000", 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.462666685, 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0074151419999999996, 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.069880+0000", 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.069880+0000", 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.069883+0000", 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.83e-06 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.069896+0000", 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2816000000000001e-05 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.069919+0000", 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2742999999999999e-05 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.069939+0000", 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0199000000000001e-05 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.070109+0000", 2024-09-16T04:02:06.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017010199999999999 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.070407+0000", 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029795700000000001 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.077296+0000", 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0068884949999999997 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 66/33 ECSubWrite(tid=100, reqid=osd.14.0:100, at_version=66'64, trim_to=0'0, roll_forward_to=65'62))", 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.723674+0000", 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.808873576, 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016421259000000001, 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723674+0000", 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723674+0000", 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723674+0000", 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9699999999999996e-07 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723731+0000", 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-16T04:02:06.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723761+0000", 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9325000000000001e-05 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723783+0000", 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2429000000000002e-05 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723985+0000", 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020243099999999999 2024-09-16T04:02:06.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724019+0000", 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3129000000000001e-05 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.738471+0000", 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014452385 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740095+0000", 2024-09-16T04:02:06.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001623869 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 66/33 ECSubWrite(tid=101, reqid=osd.14.0:101, at_version=66'66, trim_to=0'0, roll_forward_to=65'62))", 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.724901+0000", 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.807646423, 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015289522, 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724901+0000", 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724901+0000", 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724903+0000", 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9099999999999999e-06 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724955+0000", 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2420999999999998e-05 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724988+0000", 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3039e-05 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.725012+0000", 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3794999999999998e-05 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.726248+0000", 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001236516 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.726276+0000", 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8132000000000001e-05 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740105+0000", 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013828653 2024-09-16T04:02:06.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740190+0000", 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5055999999999993e-05 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s3 66/33 ECSubWrite(tid=139, reqid=osd.4.0:139, at_version=66'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.727731+0000", 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.804816216999999, 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012696292, 2024-09-16T04:02:06.591 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727731+0000", 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727731+0000", 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727731+0000", 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8599999999999998e-07 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727790+0000", 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8995e-05 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727818+0000", 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7577999999999999e-05 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727839+0000", 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1123000000000001e-05 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728012+0000", 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000172366 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728043+0000", 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1857999999999999e-05 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740357+0000", 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012313996000000001 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740427+0000", 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9789999999999994e-05 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.595 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.3s5 66/33 ECSubWrite(tid=240, reqid=osd.13.0:240, at_version=66'34, trim_to=0'0, roll_forward_to=63'30))", 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.739060+0000", 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.793486904, 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0069763029999999997, 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.739060+0000", 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.739060+0000", 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.739061+0000", 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8899999999999998e-07 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.739087+0000", 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5520999999999998e-05 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.739104+0000", 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7646e-05 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.739112+0000", 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5070000000000003e-06 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.739994+0000", 2024-09-16T04:02:06.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00088231799999999999 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740025+0000", 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0459000000000001e-05 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.745927+0000", 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0059025650000000002 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.746036+0000", 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109398 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 75/33 ECSubWrite(tid=282, reqid=client.4646.0:706, at_version=75'166, trim_to=0'0, roll_forward_to=73'164))", 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:01.036777+0000", 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.495770093, 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012394228, 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.036777+0000", 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.036777+0000", 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.036777+0000", 2024-09-16T04:02:06.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7999999999999996e-07 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041684+0000", 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049069830000000002 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041713+0000", 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8328999999999999e-05 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041736+0000", 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3693000000000002e-05 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041890+0000", 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000153351 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041916+0000", 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5695999999999999e-05 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.049069+0000", 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0071538490000000003 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.049171+0000", 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000101847 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 76/33 ECSubWrite(tid=285, reqid=client.4646.0:720, at_version=76'169, trim_to=0'0, roll_forward_to=75'166))", 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:01.751455+0000", 2024-09-16T04:02:06.604 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.7810924249999998, 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0078554030000000004, 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.751455+0000", 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.751455+0000", 2024-09-16T04:02:06.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.751456+0000", 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.756e-06 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.751923+0000", 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00046617499999999999 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.751954+0000", 2024-09-16T04:02:06.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0911999999999998e-05 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.751976+0000", 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2036e-05 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.752148+0000", 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017270499999999999 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.752167+0000", 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8811999999999999e-05 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.759202+0000", 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070345199999999998 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.759310+0000", 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010848700000000001 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.3s5 77/33 ECSubWrite(tid=270, reqid=osd.13.0:270, at_version=77'57, trim_to=0'0, roll_forward_to=73'53))", 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.736611+0000", 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.7959357850000002, 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011443514, 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.609 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.736611+0000", 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.736611+0000", 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.736613+0000", 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8199999999999999e-06 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.736678+0000", 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4372e-05 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.736714+0000", 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6565e-05 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.736743+0000", 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9263000000000001e-05 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.736902+0000", 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000158531 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.736928+0000", 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5735999999999999e-05 2024-09-16T04:02:06.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747957+0000", 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011029573000000001 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.748055+0000", 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7653999999999997e-05 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 77/33 ECSubWrite(tid=307, reqid=osd.11.0:307, at_version=77'189, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.739733+0000", 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.792814597, 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013068305000000001, 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739733+0000", 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739733+0000", 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739734+0000", 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.677e-06 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739799+0000", 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4346999999999999e-05 2024-09-16T04:02:06.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739832+0000", 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3618999999999998e-05 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739856+0000", 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3839999999999999e-05 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.740075+0000", 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021936899999999999 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.740102+0000", 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6559000000000001e-05 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.752724+0000", 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01262187 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.752801+0000", 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7024e-05 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 77/33 ECSubWrite(tid=309, reqid=osd.11.0:309, at_version=77'191, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.739936+0000", 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.7926109540000001, 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015721432, 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.618 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739936+0000", 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739936+0000", 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739937+0000", 2024-09-16T04:02:06.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739992+0000", 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4749e-05 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.740020+0000", 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8279000000000001e-05 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.740042+0000", 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1773e-05 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744030+0000", 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039875359999999999 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744054+0000", 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3989e-05 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.755523+0000", 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01146983 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.755658+0000", 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013417299999999999 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.622 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s3 77/33 ECSubWrite(tid=196, reqid=osd.4.0:193, at_version=0'0, trim_to=0'0, roll_forward_to=77'25))", 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.742522+0000", 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.7900255229999997, 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0073301420000000004, 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.742522+0000", 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.742522+0000", 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.742523+0000", 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8139999999999999e-06 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.742548+0000", 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4249000000000001e-05 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.742582+0000", 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4511999999999998e-05 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.742606+0000", 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4113000000000001e-05 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.742788+0000", 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000181268 2024-09-16T04:02:06.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.742814+0000", 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6485000000000001e-05 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.749761+0000", 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0069470069999999998 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.749852+0000", 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0693999999999996e-05 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 77/33 ECSubWrite(tid=144, reqid=client.4646.0:767, at_version=77'85, trim_to=0'0, roll_forward_to=76'83))", 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.747042+0000", 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.785505466, 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0074180319999999998, 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.627 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747042+0000", 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747042+0000", 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747043+0000", 2024-09-16T04:02:06.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1570000000000001e-06 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747181+0000", 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138615 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747211+0000", 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0012999999999999e-05 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747233+0000", 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1773e-05 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747393+0000", 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015962100000000001 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.747419+0000", 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6092999999999999e-05 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.754374+0000", 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0069550489999999996 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.754460+0000", 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5711000000000007e-05 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 77/33 ECSubWrite(tid=147, reqid=osd.14.0:147, at_version=77'89, trim_to=0'0, roll_forward_to=76'83))", 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.763278+0000", 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.7692691040000001, 2024-09-16T04:02:06.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0067525830000000004, 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.763278+0000", 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.763278+0000", 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:06.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.763279+0000", 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1829999999999999e-06 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.763320+0000", 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1139e-05 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.763344+0000", 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3379000000000001e-05 2024-09-16T04:02:06.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.763359+0000", 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5596000000000001e-05 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.763494+0000", 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013465300000000001 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.763520+0000", 2024-09-16T04:02:06.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5605e-05 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.769966+0000", 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0064468199999999998 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.770031+0000", 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4208000000000006e-05 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:06.635 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:06.636 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:06.637 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:02:06.638 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:06.633+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:06.669 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:06.669 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:06.725 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.12 primary-affinity to 0.0616457 (840402) 2024-09-16T04:02:06.731 INFO:tasks.rados.rados.0.smithi188.stdout:313: done (1 left) 2024-09-16T04:02:06.731 INFO:tasks.rados.rados.0.smithi188.stdout:314: rmattr oid 1 current snap is 32 2024-09-16T04:02:06.731 INFO:tasks.rados.rados.0.smithi188.stdout:314: done (1 left) 2024-09-16T04:02:06.731 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 48 current snap is 32 2024-09-16T04:02:06.731 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 29 v 48 (ObjNum 104 snap 22 seq_num 104) dirty exists 2024-09-16T04:02:06.732 INFO:tasks.rados.rados.0.smithi188.stdout:315: seq_num 130 ranges {4276224=638976,4915200=671744,5586944=802816,6389760=81920} 2024-09-16T04:02:06.738 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:06.733+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:06.763 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:02:06.796 INFO:tasks.rados.rados.0.smithi188.stdout:315: writing smithi18831615-48 from 4276224 to 4915200 tid 1 2024-09-16T04:02:06.802 INFO:tasks.rados.rados.0.smithi188.stdout:315: writing smithi18831615-48 from 4915200 to 5586944 tid 2 2024-09-16T04:02:06.809 INFO:tasks.rados.rados.0.smithi188.stdout:315: writing smithi18831615-48 from 5586944 to 6389760 tid 3 2024-09-16T04:02:06.811 INFO:tasks.rados.rados.0.smithi188.stdout:315: writing smithi18831615-48 from 6389760 to 6471680 tid 4 2024-09-16T04:02:06.812 INFO:tasks.rados.rados.0.smithi188.stdout:312: done (1 left) 2024-09-16T04:02:06.812 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 10 current snap is 32 2024-09-16T04:02:06.812 INFO:tasks.rados.rados.0.smithi188.stdout:315: finishing write tid 1 to smithi18831615-48 2024-09-16T04:02:06.812 INFO:tasks.rados.rados.0.smithi188.stdout:316: seq_num 131 ranges {376832=655360,1032192=425984,1458176=638976,2097152=491520} 2024-09-16T04:02:06.819 INFO:tasks.rados.rados.0.smithi188.stdout:316: writing smithi18831615-10 from 376832 to 1032192 tid 1 2024-09-16T04:02:06.822 INFO:tasks.rados.rados.0.smithi188.stdout:316: writing smithi18831615-10 from 1032192 to 1458176 tid 2 2024-09-16T04:02:06.827 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:06.827 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:06.827 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:02:06.827 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:06.828 INFO:tasks.rados.rados.0.smithi188.stdout:316: writing smithi18831615-10 from 1458176 to 2097152 tid 3 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:316: writing smithi18831615-10 from 2097152 to 2588672 tid 4 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:317: copy_from oid 9 from oid 45 current snap is 32 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:315: finishing write tid 2 to smithi18831615-48 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:315: finishing write tid 3 to smithi18831615-48 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:315: finishing write tid 4 to smithi18831615-48 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:315: finishing write tid 5 to smithi18831615-48 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:316: finishing write tid 1 to smithi18831615-10 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:315: finishing write tid 6 to smithi18831615-48 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:315: oid 48 updating version 0 to 228 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:315: oid 48 updating version 228 to 229 2024-09-16T04:02:06.832 INFO:tasks.rados.rados.0.smithi188.stdout:315: oid 48 version 229 is already newer than 223 2024-09-16T04:02:06.833 INFO:tasks.rados.rados.0.smithi188.stdout:315: oid 48 version 229 is already newer than 224 2024-09-16T04:02:06.833 INFO:tasks.rados.rados.0.smithi188.stdout:315: oid 48 version 229 is already newer than 227 2024-09-16T04:02:06.833 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 48 v 229 (ObjNum 130 snap 32 seq_num 130) dirty exists 2024-09-16T04:02:06.833 INFO:tasks.rados.rados.0.smithi188.stdout:315: left oid 48 (ObjNum 130 snap 32 seq_num 130) 2024-09-16T04:02:06.833 INFO:tasks.rados.rados.0.smithi188.stdout:315: done (2 left) 2024-09-16T04:02:06.833 INFO:tasks.rados.rados.0.smithi188.stdout:318: snap_remove snap 18 2024-09-16T04:02:06.839 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:06.837+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:06.845 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:02:06.941 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:06.937+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:07.014 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:07.015 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:07.015 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:02:07.015 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:02:07.015 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:07.029 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:02:07.042 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:02:07.037+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:02:07.143 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:07.141+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:07.158 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/33 ECSubWrite(tid=69, reqid=osd.4.0:69, at_version=41'20, trim_to=0'0, roll_forward_to=37'18))", 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.898677+0000", 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.257683727, 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0047119279999999998, 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.159 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898677+0000", 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898677+0000", 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898678+0000", 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898738+0000", 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9784999999999998e-05 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898777+0000", 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8822000000000002e-05 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898800+0000", 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3025999999999999e-05 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.899215+0000", 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041527300000000002 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.899243+0000", 2024-09-16T04:02:07.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7555e-05 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.903310+0000", 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040673369999999999 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.903389+0000", 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9396999999999999e-05 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 41/33 ECSubWrite(tid=68, reqid=osd.4.0:68, at_version=41'10, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.898853+0000", 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.257507842999999, 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0065209559999999996, 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898853+0000", 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898853+0000", 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898854+0000", 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898903+0000", 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8668000000000002e-05 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898929+0000", 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6346000000000002e-05 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.898947+0000", 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7442999999999999e-05 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.900087+0000", 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011405289999999999 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.900113+0000", 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5893000000000001e-05 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.905274+0000", 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0051606530000000003 2024-09-16T04:02:07.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.905374+0000", 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000100574 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 41/33 ECSubWrite(tid=70, reqid=osd.4.0:70, at_version=41'12, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.254758559999999, 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0095027129999999994, 2024-09-16T04:02:07.168 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901604+0000", 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.933e-06 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901660+0000", 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5979000000000001e-05 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901691+0000", 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0338999999999999e-05 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901716+0000", 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901925+0000", 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020894899999999999 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901950+0000", 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5303000000000002e-05 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.911017+0000", 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0090665969999999992 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.911105+0000", 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8153999999999997e-05 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/33 ECSubWrite(tid=71, reqid=osd.4.0:69, at_version=0'0, trim_to=0'0, roll_forward_to=41'20))", 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.907303+0000", 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.249057993999998, 2024-09-16T04:02:07.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0047894519999999996, 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907303+0000", 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907303+0000", 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907304+0000", 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.052e-06 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907317+0000", 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3019e-05 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.174 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907337+0000", 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0140999999999999e-05 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907348+0000", 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0805000000000001e-05 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907489+0000", 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014113799999999999 2024-09-16T04:02:07.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.907513+0000", 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3952e-05 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.911951+0000", 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044380519999999996 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.912093+0000", 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014129299999999999 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 44/33 ECSubWrite(tid=82, reqid=client.4646.0:352, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.928975+0000", 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.227386283000001, 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045655690000000002, 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.177 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928975+0000", 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928975+0000", 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.178 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928976+0000", 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929046+0000", 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0736999999999997e-05 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929078+0000", 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1612999999999997e-05 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929101+0000", 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3509000000000001e-05 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929281+0000", 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017924099999999999 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929307+0000", 2024-09-16T04:02:07.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6625999999999999e-05 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.933440+0000", 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041322499999999996 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.933540+0000", 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010087299999999999 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.181 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 51/33 ECSubWrite(tid=89, reqid=osd.4.0:89, at_version=51'33, trim_to=0'0, roll_forward_to=50'31))", 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:36.720301+0000", 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.436059749999998, 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0077503939999999999, 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720301+0000", 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720301+0000", 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720302+0000", 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.44e-07 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720375+0000", 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3127999999999995e-05 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720406+0000", 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0911999999999998e-05 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720428+0000", 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1776e-05 2024-09-16T04:02:07.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720652+0000", 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022443100000000001 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.720675+0000", 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.319e-05 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.727971+0000", 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072956649999999998 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:36.728052+0000", 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0648000000000004e-05 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.186 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 56/33 ECSubWrite(tid=106, reqid=osd.4.0:106, at_version=56'49, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:41.730232+0000", 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 25.426129571000001, 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053459069999999996, 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730232+0000", 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730232+0000", 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730232+0000", 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5400000000000002e-07 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730256+0000", 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4460999999999998e-05 2024-09-16T04:02:07.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730266+0000", 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8279999999999998e-06 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730277+0000", 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1089e-05 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730413+0000", 2024-09-16T04:02:07.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013620200000000001 2024-09-16T04:02:07.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.730446+0000", 2024-09-16T04:02:07.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2895999999999998e-05 2024-09-16T04:02:07.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.735500+0000", 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0050538249999999996 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:41.735577+0000", 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7352000000000001e-05 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 65/33 ECSubWrite(tid=128, reqid=client.4646.0:539, at_version=65'60, trim_to=0'0, roll_forward_to=65'58))", 2024-09-16T04:02:07.192 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.060419+0000", 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.095941995, 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0068332239999999997, 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.060419+0000", 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.060419+0000", 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.060421+0000", 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.060466+0000", 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4917999999999999e-05 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.060508+0000", 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2361999999999999e-05 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.060532+0000", 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4068999999999999e-05 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.060691+0000", 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000159269 2024-09-16T04:02:07.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.060709+0000", 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8070000000000001e-05 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.067180+0000", 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0064705530000000004 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.067252+0000", 2024-09-16T04:02:07.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2269999999999998e-05 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 66/33 ECSubWrite(tid=139, reqid=osd.4.0:139, at_version=66'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.728127+0000", 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.428233895, 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0088102679999999996, 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728127+0000", 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728127+0000", 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728128+0000", 2024-09-16T04:02:07.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1000000000000001e-06 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731766+0000", 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036377670000000001 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731816+0000", 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9607999999999997e-05 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731844+0000", 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8204999999999998e-05 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.732037+0000", 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193489 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.732062+0000", 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5029e-05 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.736621+0000", 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045590969999999998 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.736937+0000", 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031597299999999999 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/33 ECSubWrite(tid=140, reqid=osd.4.0:140, at_version=66'67, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:02:07.201 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.733453+0000", 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.422908572000001, 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0079076059999999993, 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733453+0000", 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733453+0000", 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733454+0000", 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5039999999999999e-06 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733508+0000", 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3724000000000003e-05 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733537+0000", 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9578999999999999e-05 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733559+0000", 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1826000000000001e-05 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733760+0000", 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020127099999999999 2024-09-16T04:02:07.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733774+0000", 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3952e-05 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.741301+0000", 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0075261859999999998 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.741360+0000", 2024-09-16T04:02:07.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9564000000000001e-05 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/33 ECSubWrite(tid=141, reqid=osd.4.0:141, at_version=66'69, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.733946+0000", 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.422414903, 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010060861000000001, 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733946+0000", 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733946+0000", 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733947+0000", 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.13e-06 2024-09-16T04:02:07.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.734014+0000", 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6920000000000003e-05 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.734038+0000", 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4213e-05 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.208 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.734058+0000", 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0012000000000001e-05 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.735934+0000", 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001875418 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.735958+0000", 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4589000000000001e-05 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.209 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.743961+0000", 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0080020620000000008 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.744007+0000", 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6517000000000002e-05 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/33 ECSubWrite(tid=144, reqid=osd.4.0:144, at_version=66'71, trim_to=0'0, roll_forward_to=66'69))", 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.750189+0000", 2024-09-16T04:02:07.210 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.406171759999999, 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0056554810000000004, 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.750189+0000", 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.211 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.750189+0000", 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.750191+0000", 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.663e-06 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.750429+0000", 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023825300000000001 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.750461+0000", 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1809e-05 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.750476+0000", 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5435999999999999e-05 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.750629+0000", 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015286600000000001 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.750662+0000", 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2626000000000002e-05 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.755804+0000", 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0051421410000000002 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.755845+0000", 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0686999999999999e-05 2024-09-16T04:02:07.214 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 75/33 ECSubWrite(tid=170, reqid=client.4646.0:697, at_version=75'84, trim_to=0'0, roll_forward_to=69'82))", 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:00.989037+0000", 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.16732367, 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0056185439999999996, 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.989037+0000", 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.989037+0000", 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.989039+0000", 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.66e-06 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.989858+0000", 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00081937000000000004 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.989889+0000", 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0476000000000001e-05 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.989912+0000", 2024-09-16T04:02:07.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3363e-05 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.990102+0000", 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018985700000000001 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.990129+0000", 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7339e-05 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.994549+0000", 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044192090000000003 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:00.994656+0000", 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010726999999999999 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 76/33 ECSubWrite(tid=186, reqid=client.4646.0:738, at_version=76'96, trim_to=0'0, roll_forward_to=75'94))", 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:01.815298+0000", 2024-09-16T04:02:07.219 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3410627379999998, 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054470650000000001, 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.815298+0000", 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.815298+0000", 2024-09-16T04:02:07.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.815299+0000", 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0599999999999999e-07 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.815524+0000", 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022458600000000001 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.815564+0000", 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0584999999999999e-05 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.815622+0000", 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7975e-05 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.815853+0000", 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000230996 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.815886+0000", 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2279e-05 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.820665+0000", 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0047797430000000004 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.820745+0000", 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0094999999999998e-05 2024-09-16T04:02:07.223 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 76/33 ECSubWrite(tid=188, reqid=client.4646.0:739, at_version=76'98, trim_to=0'0, roll_forward_to=76'96))", 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:01.836876+0000", 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3194852340000001, 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.006539362, 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.836876+0000", 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.836876+0000", 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.836878+0000", 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0159999999999998e-06 2024-09-16T04:02:07.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.838513+0000", 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016349800000000001 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.838546+0000", 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3458999999999999e-05 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.838564+0000", 2024-09-16T04:02:07.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7399000000000001e-05 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.838718+0000", 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000154382 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.838746+0000", 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8391999999999999e-05 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.843334+0000", 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045877629999999999 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.843415+0000", 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0970999999999994e-05 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 77/33 ECSubWrite(tid=193, reqid=osd.4.0:193, at_version=77'25, trim_to=0'0, roll_forward_to=69'23))", 2024-09-16T04:02:07.228 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.731019+0000", 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.4253421020000001, 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0074740199999999996, 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.731019+0000", 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.731019+0000", 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.731020+0000", 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.731090+0000", 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0440999999999999e-05 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.731123+0000", 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2866e-05 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.731170+0000", 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6288000000000001e-05 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732059+0000", 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00088937499999999997 2024-09-16T04:02:07.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732086+0000", 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6925999999999999e-05 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.738376+0000", 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062902610000000001 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.738493+0000", 2024-09-16T04:02:07.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116906 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 77/33 ECSubWrite(tid=194, reqid=osd.4.0:194, at_version=77'100, trim_to=0'0, roll_forward_to=76'98))", 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.732222+0000", 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.4241388769999999, 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.006458849, 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732222+0000", 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732222+0000", 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732224+0000", 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.984e-06 2024-09-16T04:02:07.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732278+0000", 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3791e-05 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732304+0000", 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6193000000000001e-05 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732327+0000", 2024-09-16T04:02:07.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2532999999999999e-05 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732499+0000", 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017232700000000001 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.732526+0000", 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6923e-05 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.738585+0000", 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060585869999999998 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.738681+0000", 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6510999999999998e-05 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 77/33 ECSubWrite(tid=195, reqid=osd.4.0:195, at_version=77'102, trim_to=0'0, roll_forward_to=76'98))", 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.734821+0000", 2024-09-16T04:02:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.4215399690000003, 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0089874410000000005, 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734821+0000", 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734821+0000", 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734823+0000", 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734891+0000", 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7774000000000005e-05 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734920+0000", 2024-09-16T04:02:07.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9549000000000001e-05 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734946+0000", 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5732999999999999e-05 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.735341+0000", 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039543 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.735365+0000", 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3543000000000001e-05 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.743728+0000", 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0083628179999999993 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.743809+0000", 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0900000000000001e-05 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.241 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.es1 77/33 ECSubWrite(tid=27, reqid=osd.2.0:27, at_version=77'10, trim_to=0'0, roll_forward_to=73'8))", 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.744000+0000", 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.412361164, 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072135419999999999, 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744000+0000", 2024-09-16T04:02:07.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744000+0000", 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744001+0000", 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0999999999999998e-07 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744058+0000", 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7334999999999999e-05 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744089+0000", 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1118999999999998e-05 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744123+0000", 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3775000000000001e-05 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744429+0000", 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00030627399999999999 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744454+0000", 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4643000000000001e-05 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.751097+0000", 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0066430459999999997 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.751213+0000", 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011674 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 81/33 ECSubWrite(tid=217, reqid=osd.4.0:217, at_version=81'114, trim_to=0'0, roll_forward_to=79'112))", 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:06.749966+0000", 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.40639502199999999, 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0050595969999999999, 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.246 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.749966+0000", 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.749966+0000", 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.749966+0000", 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0600000000000001e-07 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.749989+0000", 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2484e-05 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.750000+0000", 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0577999999999999e-05 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.750008+0000", 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2660000000000001e-06 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.750114+0000", 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010587800000000001 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.750129+0000", 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5175e-05 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.754969+0000", 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048403400000000003 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:06.755026+0000", 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.647e-05 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.250 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:07.252 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:02:07.253 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:02:07.249+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:02:07.353 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:07.352+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:07.384 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:07.384 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:07.384 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:02:07.384 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:07.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:02:07.455 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:07.453+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:07.518 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:07.519 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:07.519 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:02:07.519 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:02:07.519 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:07.533 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:02:07.582 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:07.581+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:07.657 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:07.653+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:07.663 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 69.19825865, 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.56051925899999999, 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460473+0000", 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.097e-06 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460488+0000", 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5335000000000001e-05 2024-09-16T04:02:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460512+0000", 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4108999999999999e-05 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460533+0000", 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0721999999999999e-05 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460655+0000", 2024-09-16T04:02:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122328 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460689+0000", 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014402+0000", 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.55371241699999996 2024-09-16T04:02:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014725+0000", 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000323345 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.015144+0000", 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041881500000000001 2024-09-16T04:02:07.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020086+0000", 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049418869999999998 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020836+0000", 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074984300000000004 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020926+0000", 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0070999999999999e-05 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020990+0000", 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4256000000000003e-05 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.671 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.245989672999997, 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019211259000000001, 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 117 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.672 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412741+0000", 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8700000000000005e-07 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412747+0000", 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1929999999999998e-06 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412757+0000", 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0039e-05 2024-09-16T04:02:07.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412764+0000", 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3259999999999998e-06 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426386+0000", 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013621668999999999 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426547+0000", 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000161192 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426753+0000", 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020624599999999999 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428173+0000", 2024-09-16T04:02:07.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001419305 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431799+0000", 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036262320000000001 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431890+0000", 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0987000000000001e-05 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431951+0000", 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1383000000000005e-05 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.893163950000002, 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016729104000000002, 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.678 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 160 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765566+0000", 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765567+0000", 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765573+0000", 2024-09-16T04:02:07.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0260000000000004e-06 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765583+0000", 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765590+0000", 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4800000000000004e-06 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774469+0000", 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.008879184 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774666+0000", 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019695699999999999 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.774998+0000", 2024-09-16T04:02:07.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00033151099999999999 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778132+0000", 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031338849999999999 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782066+0000", 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039342630000000003 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782181+0000", 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000115093 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782295+0000", 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113726 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.684 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.893127206000003, 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017211085000000001, 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 161 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765603+0000", 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:02:07.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765727+0000", 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124184 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765750+0000", 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2296000000000001e-05 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765759+0000", 2024-09-16T04:02:07.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2760000000000006e-06 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775766+0000", 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010007085000000001 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775856+0000", 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9683000000000002e-05 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.776003+0000", 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014695999999999999 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779325+0000", 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033221930000000002 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782596+0000", 2024-09-16T04:02:07.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032713519999999999 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782726+0000", 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013011500000000001 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782813+0000", 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7174000000000003e-05 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.690 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.892952804000004, 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017217072999999999, 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 162 2024-09-16T04:02:07.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765778+0000", 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765784+0000", 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2659999999999998e-06 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765794+0000", 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.029e-05 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765809+0000", 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4474999999999999e-05 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777438+0000", 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011628929 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777529+0000", 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1577999999999996e-05 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777666+0000", 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013660699999999999 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781937+0000", 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042713660000000004 2024-09-16T04:02:07.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782885+0000", 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00094829699999999996 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782947+0000", 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1261999999999997e-05 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782994+0000", 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7139999999999999e-05 2024-09-16T04:02:07.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.892902421000002, 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01977027, 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.697 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 163 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:02:07.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765828+0000", 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765834+0000", 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7830000000000004e-06 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765844+0000", 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0149000000000001e-05 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765852+0000", 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0199999999999994e-06 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778628+0000", 2024-09-16T04:02:07.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012776183999999999 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778715+0000", 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6941000000000007e-05 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778850+0000", 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013517800000000001 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783007+0000", 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041571960000000002 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785458+0000", 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002450626 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785539+0000", 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1213999999999999e-05 2024-09-16T04:02:07.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785597+0000", 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8258999999999999e-05 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.703 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.892856743999999, 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019887134000000001, 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 164 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765874+0000", 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0130000000000001e-06 2024-09-16T04:02:07.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765996+0000", 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122039 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766015+0000", 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9405000000000001e-05 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766025+0000", 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0096e-05 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779427+0000", 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013401569 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779649+0000", 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022229500000000001 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.780098+0000", 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00044856699999999998 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783018+0000", 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002919792 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785656+0000", 2024-09-16T04:02:07.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026383489999999999 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785714+0000", 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8177000000000002e-05 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785760+0000", 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5831999999999998e-05 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.709 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.892685822000004, 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021760347999999999, 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.710 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 165 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766045+0000", 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766051+0000", 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8140000000000003e-06 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766062+0000", 2024-09-16T04:02:07.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0592e-05 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766069+0000", 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3629999999999996e-06 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781249+0000", 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015179349999999999 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781356+0000", 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000107747 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781496+0000", 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013985500000000001 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783295+0000", 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017986199999999999 2024-09-16T04:02:07.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787679+0000", 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0043837559999999999 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787752+0000", 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3192e-05 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787804+0000", 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2182999999999998e-05 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.723303079999994, 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016713338000000001, 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.716 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 189 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935426+0000", 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935428+0000", 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935443+0000", 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4599e-05 2024-09-16T04:02:07.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935468+0000", 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4788000000000001e-05 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.935486+0000", 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8249000000000001e-05 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949701+0000", 2024-09-16T04:02:07.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014214839 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.949818+0000", 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011776499999999999 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.950001+0000", 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018260100000000001 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.951914+0000", 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001913049 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952007+0000", 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3006999999999999e-05 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952083+0000", 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5722000000000004e-05 2024-09-16T04:02:07.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.952140+0000", 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7006000000000003e-05 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:347 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.735748571999999, 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018664877, 2024-09-16T04:02:07.722 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 347 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922982+0000", 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.457e-06 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922991+0000", 2024-09-16T04:02:07.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923032+0000", 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0878999999999999e-05 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923051+0000", 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9406e-05 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923225+0000", 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000173781 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923244+0000", 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8583000000000001e-05 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928577+0000", 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053332600000000003 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928836+0000", 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025873200000000002 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.941646+0000", 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01281 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:348 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:07.727 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.735615393000003, 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020600073999999999, 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 348 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923116+0000", 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8530000000000001e-06 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923127+0000", 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1396000000000001e-05 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923152+0000", 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4253e-05 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923168+0000", 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6685999999999999e-05 2024-09-16T04:02:07.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923332+0000", 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016361899999999999 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923353+0000", 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0894999999999999e-05 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929889+0000", 2024-09-16T04:02:07.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0065358930000000001 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.930145+0000", 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000256452 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.943714+0000", 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013569027000000001 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:477 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.916790394, 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017902273, 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 477 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.733 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741942+0000", 2024-09-16T04:02:07.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8830000000000002e-06 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741957+0000", 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4922999999999999e-05 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741991+0000", 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4094999999999998e-05 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742009+0000", 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7958999999999999e-05 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742143+0000", 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013377699999999999 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742155+0000", 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2692e-05 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.747863+0000", 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057079360000000003 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.748115+0000", 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025199499999999998 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.759841+0000", 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011726013 2024-09-16T04:02:07.737 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:478 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.916678823000002, 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020150529, 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.738 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 478 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742052+0000", 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.423e-06 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742061+0000", 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8370000000000003e-06 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742084+0000", 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3034999999999999e-05 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742100+0000", 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5798999999999999e-05 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742319+0000", 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000218789 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742340+0000", 2024-09-16T04:02:07.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1069000000000001e-05 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.750020+0000", 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0076799820000000001 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.750277+0000", 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025761000000000002 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.762201+0000", 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011923985 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:479 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.916596807000001, 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021828410999999999, 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.743 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 479 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742134+0000", 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2929999999999999e-06 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742143+0000", 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7569999999999993e-06 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742163+0000", 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9839000000000001e-05 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742176+0000", 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3782e-05 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742460+0000", 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00028378600000000001 2024-09-16T04:02:07.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742471+0000", 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0856000000000001e-05 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.753628+0000", 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011156981999999999 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.753886+0000", 2024-09-16T04:02:07.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025786400000000001 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.763961+0000", 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010075252 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:612 3.3s0 3:c7e4f386:::smithi18831615-44:head [append 0~655360] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.917245+0000", 2024-09-16T04:02:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.741484986, 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020015294999999999, 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 612 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.917245+0000", 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.917245+0000", 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.917248+0000", 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2789999999999999e-06 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918441+0000", 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011929919999999999 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918481+0000", 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0342000000000001e-05 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918499+0000", 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7396000000000001e-05 2024-09-16T04:02:07.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918669+0000", 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017088699999999999 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918683+0000", 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3613e-05 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.923229+0000", 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045464520000000003 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.923402+0000", 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000172786 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.925580+0000", 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021778050000000001 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.936521+0000", 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010940938000000001 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.937081+0000", 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00055951899999999997 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.937260+0000", 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017928599999999999 2024-09-16T04:02:07.754 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:614 3.3s0 3:c7e4f386:::smithi18831615-44:head [append 0~475136] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.926452+0000", 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.732277364, 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017618722999999999, 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 614 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.926452+0000", 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.926452+0000", 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.926454+0000", 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9700000000000002e-06 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927237+0000", 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078264100000000004 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927270+0000", 2024-09-16T04:02:07.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3569000000000003e-05 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927293+0000", 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2578999999999998e-05 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.932624+0000", 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053313350000000004 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.932929+0000", 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000304739 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.935075+0000", 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021457989999999999 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.938167+0000", 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00309254 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.943845+0000", 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0056773860000000004 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.944071+0000", 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022616499999999999 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:615 3.3s0 3:c7e4f386:::smithi18831615-44:head [append 0~81920] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.927365+0000", 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.731365018, 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018348695000000002, 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 615 2024-09-16T04:02:07.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927365+0000", 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927365+0000", 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927367+0000", 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.103e-06 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927547+0000", 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018066999999999999 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927581+0000", 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3782000000000003e-05 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927599+0000", 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8133000000000001e-05 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.938506+0000", 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010906735000000001 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.938732+0000", 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022580500000000001 2024-09-16T04:02:07.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.939846+0000", 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011143679999999999 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.944308+0000", 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044614650000000004 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.945525+0000", 2024-09-16T04:02:07.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012174270000000001 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.945713+0000", 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000188207 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:616 3.3s0 3:c7e4f386:::smithi18831615-44:head [setxattr _header (57) in=7b] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.927636+0000", 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.731093411, 2024-09-16T04:02:07.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021030818999999999, 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 616 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927636+0000", 2024-09-16T04:02:07.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927636+0000", 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927638+0000", 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927648+0000", 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8630000000000003e-06 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927669+0000", 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1563e-05 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927689+0000", 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9998999999999999e-05 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.940931+0000", 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013242017 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.941083+0000", 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015142700000000001 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.942463+0000", 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001380582 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.944932+0000", 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0024683270000000002 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.948416+0000", 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034847419999999999 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.948667+0000", 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000250633 2024-09-16T04:02:07.771 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:617 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.927718+0000", 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.731011848, 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026128365000000001, 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.772 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 617 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927718+0000", 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927718+0000", 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927719+0000", 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4839999999999999e-06 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927727+0000", 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5989999999999996e-06 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927752+0000", 2024-09-16T04:02:07.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5398999999999999e-05 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927772+0000", 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0256000000000001e-05 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.946010+0000", 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018237386000000001 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.946244+0000", 2024-09-16T04:02:07.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023392399999999999 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.949754+0000", 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035106040000000001 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.949954+0000", 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000199913 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.953846+0000", 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0038917999999999999 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:705 3.2s0 3:45f0f92f:::smithi18831615-17:11 [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:01.000458+0000", 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.6582718390000002, 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018396058, 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:07.777 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 705 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000458+0000", 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000458+0000", 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000459+0000", 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000468+0000", 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0529999999999996e-06 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000490+0000", 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2382999999999999e-05 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000505+0000", 2024-09-16T04:02:07.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5009e-05 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000914+0000", 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000408737 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000933+0000", 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9286000000000001e-05 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:07.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.006979+0000", 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060450290000000004 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.007138+0000", 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015961400000000001 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.018854+0000", 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011715621000000001 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:07.783 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:07.784 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-09-16T04:02:07.785 INFO:tasks.rados.rados.0.smithi188.stdout:318: done (2 left) 2024-09-16T04:02:07.785 INFO:tasks.rados.rados.0.smithi188.stdout:319: read oid 18 snap -1 2024-09-16T04:02:07.785 INFO:tasks.rados.rados.0.smithi188.stdout:319: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:02:07.785 INFO:tasks.rados.rados.0.smithi188.stdout:316: finishing write tid 2 to smithi18831615-10 2024-09-16T04:02:07.785 INFO:tasks.rados.rados.0.smithi188.stdout:316: finishing write tid 3 to smithi18831615-10 2024-09-16T04:02:07.785 INFO:tasks.rados.rados.0.smithi188.stdout:316: finishing write tid 4 to smithi18831615-10 2024-09-16T04:02:07.785 INFO:tasks.rados.rados.0.smithi188.stdout:316: finishing write tid 5 to smithi18831615-10 2024-09-16T04:02:07.785 INFO:tasks.rados.rados.0.smithi188.stdout:316: finishing write tid 6 to smithi18831615-10 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:316: oid 10 updating version 0 to 55 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:316: oid 10 updating version 55 to 58 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:316: oid 10 version 58 is already newer than 57 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:316: oid 10 version 58 is already newer than 54 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:316: oid 10 version 58 is already newer than 56 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 10 v 58 (ObjNum 131 snap 32 seq_num 131) dirty exists 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:316: left oid 10 (ObjNum 131 snap 32 seq_num 131) 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:317: finishing copy_from racing read to smithi18831615-9 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:317: finishing copy_from to smithi18831615-9 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 9 v 26 (ObjNum 86 snap 12 seq_num 86) dirty exists 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:316: done (2 left) 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:317: done (1 left) 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 7 current snap is 32 2024-09-16T04:02:07.786 INFO:tasks.rados.rados.0.smithi188.stdout:320: seq_num 132 ranges {114688=671744,786432=573440,1359872=753664,2113536=229376} 2024-09-16T04:02:07.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:02:07.787 INFO:tasks.rados.rados.0.smithi188.stdout:320: writing smithi18831615-7 from 114688 to 786432 tid 1 2024-09-16T04:02:07.788 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:07.781+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:07.791 INFO:tasks.rados.rados.0.smithi188.stdout:320: writing smithi18831615-7 from 786432 to 1359872 tid 2 2024-09-16T04:02:07.799 INFO:tasks.rados.rados.0.smithi188.stdout:320: writing smithi18831615-7 from 1359872 to 2113536 tid 3 2024-09-16T04:02:07.802 INFO:tasks.rados.rados.0.smithi188.stdout:320: writing smithi18831615-7 from 2113536 to 2342912 tid 4 2024-09-16T04:02:07.802 INFO:tasks.rados.rados.0.smithi188.stdout:321: rmattr oid 3 current snap is 32 2024-09-16T04:02:07.802 INFO:tasks.rados.rados.0.smithi188.stdout:321: done (2 left) 2024-09-16T04:02:07.802 INFO:tasks.rados.rados.0.smithi188.stdout:322: snap_create 2024-09-16T04:02:07.819 INFO:tasks.rados.rados.0.smithi188.stdout:320: finishing write tid 1 to smithi18831615-7 2024-09-16T04:02:07.822 INFO:tasks.rados.rados.0.smithi188.stdout:319: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:02:07.885 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:07.881+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:07.922 INFO:tasks.rados.rados.0.smithi188.stdout:320: finishing write tid 2 to smithi18831615-7 2024-09-16T04:02:07.922 INFO:tasks.rados.rados.0.smithi188.stdout:320: finishing write tid 3 to smithi18831615-7 2024-09-16T04:02:07.922 INFO:tasks.rados.rados.0.smithi188.stdout:320: finishing write tid 4 to smithi18831615-7 2024-09-16T04:02:07.922 INFO:tasks.rados.rados.0.smithi188.stdout:320: finishing write tid 5 to smithi18831615-7 2024-09-16T04:02:07.922 INFO:tasks.rados.rados.0.smithi188.stdout:320: finishing write tid 6 to smithi18831615-7 2024-09-16T04:02:07.922 INFO:tasks.rados.rados.0.smithi188.stdout:320: oid 7 updating version 0 to 81 2024-09-16T04:02:07.923 INFO:tasks.rados.rados.0.smithi188.stdout:320: oid 7 version 81 is already newer than 80 2024-09-16T04:02:07.923 INFO:tasks.rados.rados.0.smithi188.stdout:320: oid 7 version 81 is already newer than 77 2024-09-16T04:02:07.923 INFO:tasks.rados.rados.0.smithi188.stdout:320: oid 7 version 81 is already newer than 78 2024-09-16T04:02:07.923 INFO:tasks.rados.rados.0.smithi188.stdout:320: oid 7 version 81 is already newer than 79 2024-09-16T04:02:07.923 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 7 v 81 (ObjNum 132 snap 32 seq_num 132) dirty exists 2024-09-16T04:02:07.923 INFO:tasks.rados.rados.0.smithi188.stdout:320: left oid 7 (ObjNum 132 snap 32 seq_num 132) 2024-09-16T04:02:07.923 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:07.924 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:07.924 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:02:07.924 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:07.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:02:07.986 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:07.985+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:08.062 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:08.062 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:08.062 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:02:08.062 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:02:08.063 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:08.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:02:08.087 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:08.085+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:08.189 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:08.184+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:08.200 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:08.200 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:02:08.200 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:02:08.200 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:02:08.200 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:607 3.es0 3:731b4600:::smithi18831615-23:head [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:55.898089+0000", 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 12.299727001000001, 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010252327, 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 607 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.201 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898089+0000", 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898089+0000", 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.202 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898090+0000", 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8400000000000002e-07 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898096+0000", 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.0530000000000003e-06 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.203 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898111+0000", 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5352000000000001e-05 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898120+0000", 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.8899999999999996e-06 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898369+0000", 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000249421 2024-09-16T04:02:08.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898380+0000", 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0613e-05 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.902813+0000", 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0044335349999999997 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.205 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.903049+0000", 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000235911 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.908341+0000", 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.005291568 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.206 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:608 3.es0 3:731b4600:::smithi18831615-23:head [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:55.898198+0000", 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 12.299617374, 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012172129, 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 608 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.207 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898198+0000", 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898198+0000", 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898201+0000", 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4499999999999998e-06 2024-09-16T04:02:08.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898214+0000", 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3733999999999999e-05 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898243+0000", 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8544999999999999e-05 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898261+0000", 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8349999999999999e-05 2024-09-16T04:02:08.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898441+0000", 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000179882 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898451+0000", 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8730000000000002e-06 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.903831+0000", 2024-09-16T04:02:08.210 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053801359999999998 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904520+0000", 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00068860700000000004 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.910370+0000", 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058505520000000002 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.211 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:765 3.es0 3:74948e3e:::smithi18831615-25:head [read 0~1638400,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:01.942890+0000", 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 6.254926019, 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015336928999999999, 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:08.212 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 765 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942890+0000", 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942890+0000", 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.213 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942891+0000", 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.297e-06 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942898+0000", 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4229999999999999e-06 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.214 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.216 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942916+0000", 2024-09-16T04:02:08.216 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7516e-05 2024-09-16T04:02:08.216 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.216 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.216 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942930+0000", 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3845999999999999e-05 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.952436+0000", 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0095067669999999993 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.952686+0000", 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00024933099999999999 2024-09-16T04:02:08.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.958226+0000", 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055407490000000002 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.3s4 77/33 ECSubWrite(tid=270, reqid=osd.13.0:270, at_version=77'57, trim_to=0'0, roll_forward_to=73'53))", 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:02.735562+0000", 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.46225352, 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010534045000000001, 2024-09-16T04:02:08.218 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.735562+0000", 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.735562+0000", 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.219 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.735564+0000", 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9400000000000001e-06 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.735611+0000", 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.6805999999999999e-05 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.220 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.735636+0000", 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5032e-05 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.735656+0000", 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9792999999999998e-05 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.736382+0000", 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00072615199999999998 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.221 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.736406+0000", 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4128999999999999e-05 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.746020+0000", 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0096138950000000008 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.746096+0000", 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6297999999999994e-05 2024-09-16T04:02:08.222 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.4s1 81/33 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=81'50, trim_to=0'0, roll_forward_to=80'48))", 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.735794+0000", 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.4620220779999999, 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.025016644000000001, 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.223 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735794+0000", 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735794+0000", 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735795+0000", 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7060000000000001e-06 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.224 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735854+0000", 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8462999999999999e-05 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735886+0000", 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2058999999999999e-05 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735908+0000", 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2353000000000001e-05 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.225 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738447+0000", 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0025387460000000001 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738467+0000", 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0635000000000001e-05 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.760710+0000", 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.022242893999999999 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.760810+0000", 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.9788000000000001e-05 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.4s1 81/33 ECSubWrite(tid=82, reqid=osd.8.0:82, at_version=81'52, trim_to=0'0, roll_forward_to=80'48))", 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.738367+0000", 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.4594484599999999, 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.032503008, 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738367+0000", 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.228 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738367+0000", 2024-09-16T04:02:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738368+0000", 2024-09-16T04:02:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2500000000000001e-06 2024-09-16T04:02:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738418+0000", 2024-09-16T04:02:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.9885000000000001e-05 2024-09-16T04:02:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738450+0000", 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1633e-05 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738517+0000", 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7151999999999997e-05 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740733+0000", 2024-09-16T04:02:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0022160420000000001 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740757+0000", 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3750000000000001e-05 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770786+0000", 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.030029160999999999 2024-09-16T04:02:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770870+0000", 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.4135000000000007e-05 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs1 81/33 ECSubWrite(tid=334, reqid=osd.11.0:334, at_version=81'68, trim_to=0'0, roll_forward_to=77'66))", 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.739676+0000", 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.458139181, 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.028451536999999999, 2024-09-16T04:02:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739676+0000", 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739676+0000", 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.234 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739678+0000", 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.916e-06 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739737+0000", 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8659000000000002e-05 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739789+0000", 2024-09-16T04:02:08.235 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.1949e-05 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739812+0000", 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3553e-05 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740209+0000", 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000396463 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.236 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740234+0000", 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5026000000000001e-05 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768051+0000", 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.027817172000000001 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768128+0000", 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6798999999999994e-05 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.237 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.238 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.238 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.238 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.238 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.3s4 81/33 ECSubWrite(tid=281, reqid=osd.13.0:281, at_version=81'59, trim_to=0'0, roll_forward_to=77'57))", 2024-09-16T04:02:08.238 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.741148+0000", 2024-09-16T04:02:08.238 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.456667559, 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.026862554, 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741148+0000", 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.239 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741148+0000", 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741149+0000", 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.4300000000000003e-07 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.240 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741201+0000", 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2519000000000003e-05 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741225+0000", 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4247e-05 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741247+0000", 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1149e-05 2024-09-16T04:02:08.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741399+0000", 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015270400000000001 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741423+0000", 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4070000000000002e-05 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.767907+0000", 2024-09-16T04:02:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.026483754000000002 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768011+0000", 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010346800000000001 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.3s4 81/33 ECSubWrite(tid=282, reqid=osd.13.0:282, at_version=81'61, trim_to=0'0, roll_forward_to=77'57))", 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.742752+0000", 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.455063432, 2024-09-16T04:02:08.243 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.028545241999999998, 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742752+0000", 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742752+0000", 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742754+0000", 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7400000000000001e-06 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742807+0000", 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2961999999999999e-05 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742849+0000", 2024-09-16T04:02:08.245 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.2329000000000001e-05 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742873+0000", 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3883000000000002e-05 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755109+0000", 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012235721 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.246 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755134+0000", 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4805999999999999e-05 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771234+0000", 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.016100224999999999 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771297+0000", 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.3576000000000003e-05 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 81/33 ECSubWrite(tid=155, reqid=osd.14.0:155, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.747358+0000", 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.450458032, 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020922782000000001, 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747358+0000", 2024-09-16T04:02:08.248 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747358+0000", 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747358+0000", 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747413+0000", 2024-09-16T04:02:08.249 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.4619e-05 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747442+0000", 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9422000000000001e-05 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747464+0000", 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.179e-05 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.250 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747633+0000", 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016887 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747668+0000", 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5203000000000002e-05 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768152+0000", 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020483627000000001 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.251 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768280+0000", 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012855399999999999 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 81/33 ECSubWrite(tid=156, reqid=osd.14.0:156, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.749686+0000", 2024-09-16T04:02:08.252 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.4481294820000001, 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021277265, 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749686+0000", 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.253 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749686+0000", 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749688+0000", 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7060000000000001e-06 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749738+0000", 2024-09-16T04:02:08.254 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0303000000000003e-05 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749777+0000", 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8399000000000003e-05 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749800+0000", 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3416000000000001e-05 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.255 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755725+0000", 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059249259999999996 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755752+0000", 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7219000000000001e-05 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.256 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770896+0000", 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015144316999999999 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770963+0000", 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6978999999999997e-05 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.257 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s2 81/33 ECSubWrite(tid=28, reqid=osd.5.0:28, at_version=81'23, trim_to=0'0, roll_forward_to=79'21))", 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.751718+0000", 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.4460975199999999, 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019320789000000001, 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.258 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751718+0000", 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751718+0000", 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.259 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751719+0000", 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.4600000000000004e-07 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751791+0000", 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.2033000000000006e-05 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751824+0000", 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3129000000000001e-05 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.260 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751846+0000", 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.262e-05 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755683+0000", 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0038367679999999999 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.261 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755706+0000", 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2829000000000001e-05 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770982+0000", 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015275739999999999 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771039+0000", 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7024000000000002e-05 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.262 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs1 81/33 ECSubWrite(tid=336, reqid=osd.11.0:336, at_version=81'70, trim_to=0'0, roll_forward_to=77'66))", 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.753557+0000", 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.444258781, 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.017652517999999999, 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.263 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753557+0000", 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753557+0000", 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.264 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753559+0000", 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8500000000000001e-06 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753636+0000", 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7095000000000006e-05 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.265 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753666+0000", 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0406e-05 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753688+0000", 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1416e-05 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755220+0000", 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001531988 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755245+0000", 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4983000000000001e-05 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.267 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771138+0000", 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015893442000000001 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771209+0000", 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1338e-05 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s5 81/33 ECSubWrite(tid=222, reqid=osd.4.0:222, at_version=81'33, trim_to=0'0, roll_forward_to=79'31))", 2024-09-16T04:02:08.268 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.763721+0000", 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.434094424, 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013019978, 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763721+0000", 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763721+0000", 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763723+0000", 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9099999999999999e-06 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763799+0000", 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6068000000000004e-05 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.270 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763840+0000", 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.0720000000000003e-05 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763862+0000", 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2538999999999999e-05 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.764054+0000", 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019151300000000001 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.271 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.764084+0000", 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0583000000000002e-05 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.776658+0000", 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012573457999999999 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.776741+0000", 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.3187000000000002e-05 2024-09-16T04:02:08.272 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s5 81/33 ECSubWrite(tid=224, reqid=osd.4.0:224, at_version=81'35, trim_to=0'0, roll_forward_to=79'31))", 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.766937+0000", 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 1.430879035, 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013032854, 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.273 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.766937+0000", 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.766937+0000", 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.766938+0000", 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.886e-06 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.274 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.766999+0000", 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.0232999999999997e-05 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.767044+0000", 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.5785000000000003e-05 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.767067+0000", 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2744000000000001e-05 2024-09-16T04:02:08.275 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770282+0000", 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0032146200000000001 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770308+0000", 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6171999999999999e-05 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.779896+0000", 2024-09-16T04:02:08.276 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0095884730000000001 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.779969+0000", 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.2941000000000005e-05 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.3s4 82/33 ECSubWrite(tid=285, reqid=osd.13.0:285, at_version=82'63, trim_to=0'0, roll_forward_to=81'61))", 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.774113+0000", 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.42370299099999997, 2024-09-16T04:02:08.277 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014629262000000001, 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774113+0000", 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774113+0000", 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.278 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774113+0000", 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.9699999999999996e-07 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774169+0000", 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5522000000000001e-05 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.279 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774194+0000", 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5032999999999999e-05 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774215+0000", 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1650000000000001e-05 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774423+0000", 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000207395 2024-09-16T04:02:08.280 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774448+0000", 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5259999999999999e-05 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788647+0000", 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014198489999999999 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788742+0000", 2024-09-16T04:02:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5315000000000005e-05 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s5 82/33 ECSubWrite(tid=233, reqid=osd.4.0:233, at_version=82'39, trim_to=0'0, roll_forward_to=81'35))", 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.775012+0000", 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.42280343500000001, 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013457884999999999, 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.282 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775012+0000", 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775012+0000", 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775014+0000", 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6559999999999999e-06 2024-09-16T04:02:08.283 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775067+0000", 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2862999999999999e-05 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775098+0000", 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1272000000000001e-05 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775119+0000", 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1290000000000001e-05 2024-09-16T04:02:08.284 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775326+0000", 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00020677900000000001 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775351+0000", 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5219e-05 2024-09-16T04:02:08.285 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788373+0000", 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013021721 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788470+0000", 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.7084999999999996e-05 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.286 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 82/33 ECSubWrite(tid=164, reqid=osd.14.0:164, at_version=82'106, trim_to=0'0, roll_forward_to=81'104))", 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.778408+0000", 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.41940760599999999, 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010133631000000001, 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.778408+0000", 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.778408+0000", 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.778409+0000", 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.347e-06 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.778462+0000", 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2389000000000003e-05 2024-09-16T04:02:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.778491+0000", 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9206000000000001e-05 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.778556+0000", 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.4754999999999999e-05 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.778779+0000", 2024-09-16T04:02:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00022367900000000001 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.778805+0000", 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5899999999999999e-05 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788449+0000", 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0096433400000000002 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788542+0000", 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.3015000000000003e-05 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs1 82/33 ECSubWrite(tid=355, reqid=osd.11.0:355, at_version=82'74, trim_to=0'0, roll_forward_to=81'72))", 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.793647+0000", 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.404168623, 2024-09-16T04:02:08.291 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012159118999999999, 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.793647+0000", 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.292 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.793647+0000", 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.793649+0000", 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8500000000000001e-06 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.793704+0000", 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5655000000000002e-05 2024-09-16T04:02:08.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.793734+0000", 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9162999999999998e-05 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.793754+0000", 2024-09-16T04:02:08.294 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0820000000000001e-05 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.793976+0000", 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000221852 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.794002+0000", 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5663000000000001e-05 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.295 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.805760+0000", 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011758457999999999 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.805806+0000", 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.5658000000000002e-05 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 82/33 ECSubWrite(tid=169, reqid=osd.14.0:169, at_version=82'110, trim_to=0'0, roll_forward_to=82'108))", 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.801920+0000", 2024-09-16T04:02:08.296 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.39589598199999998, 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010488610000000001, 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801920+0000", 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801920+0000", 2024-09-16T04:02:08.297 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801921+0000", 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.626e-06 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801971+0000", 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.9743000000000001e-05 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.802005+0000", 2024-09-16T04:02:08.298 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4038999999999997e-05 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.802027+0000", 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1622999999999999e-05 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.802304+0000", 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00027744400000000001 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.299 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.802328+0000", 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3730000000000001e-05 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.812361+0000", 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010032964 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.812408+0000", 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7441000000000002e-05 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.300 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.301 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.301 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.301 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:08.302 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:08.300+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:08.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:02:08.403 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:08.400+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:08.431 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:08.431 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:08.431 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:02:08.431 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:08.444 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:02:08.504 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:02:08.500+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:02:08.563 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:08.563 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:08.563 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:02:08.563 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:02:08.563 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:08.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:02:08.605 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:08.601+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:08.693 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:02:08.699 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:08.699 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:02:08.699 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:02:08.699 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:02:08.700 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.700 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 41/33 ECSubWrite(tid=101, reqid=osd.11.0:101, at_version=41'54, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:02:08.700 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:02:08.700 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 41.778871764000002, 2024-09-16T04:02:08.700 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011980414999999999, 2024-09-16T04:02:08.700 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.701 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.701 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.701 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.701 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.701 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.702 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:02:08.702 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.702 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.702 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916370+0000", 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916371+0000", 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6300000000000004e-07 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916424+0000", 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2497999999999998e-05 2024-09-16T04:02:08.703 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916451+0000", 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7673000000000001e-05 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916473+0000", 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1573000000000002e-05 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.704 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916863+0000", 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00039059000000000001 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916888+0000", 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4629000000000001e-05 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928269+0000", 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011381092000000001 2024-09-16T04:02:08.705 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928351+0000", 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1596999999999998e-05 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 41/33 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=41'56, trim_to=0'0, roll_forward_to=40'52))", 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 41.778713508999999, 2024-09-16T04:02:08.706 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015945876000000001, 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916529+0000", 2024-09-16T04:02:08.707 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916530+0000", 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.133e-06 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916571+0000", 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1539000000000003e-05 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.708 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916594+0000", 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2915999999999999e-05 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916613+0000", 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8766e-05 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918993+0000", 2024-09-16T04:02:08.709 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0023803050000000001 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919007+0000", 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3703000000000001e-05 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932390+0000", 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013382897 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.710 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932474+0000", 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.4617000000000007e-05 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.as5 41/33 ECSubWrite(tid=13, reqid=osd.3.0:13, at_version=41'8, trim_to=0'0, roll_forward_to=38'6))", 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 41.777989726000001, 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010948411999999999, 2024-09-16T04:02:08.711 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.712 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917252+0000", 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917253+0000", 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7299999999999996e-07 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917309+0000", 2024-09-16T04:02:08.713 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5988e-05 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917340+0000", 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1421999999999998e-05 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917364+0000", 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3893e-05 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.714 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917525+0000", 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016099500000000001 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.917551+0000", 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5344999999999999e-05 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928110+0000", 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010559192 2024-09-16T04:02:08.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.928201+0000", 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.1003999999999998e-05 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/33 ECSubWrite(tid=197, reqid=osd.13.0:197, at_version=41'18, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 41.776171636000001, 2024-09-16T04:02:08.716 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013473318999999999, 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.717 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919070+0000", 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919071+0000", 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7e-07 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919120+0000", 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.9595000000000002e-05 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.718 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919147+0000", 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6415999999999999e-05 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919176+0000", 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9002000000000001e-05 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919332+0000", 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015599200000000001 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.719 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.919355+0000", 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3312e-05 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932433+0000", 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013077961000000001 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.932544+0000", 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000110571 2024-09-16T04:02:08.720 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/33 ECSubWrite(tid=199, reqid=osd.13.0:199, at_version=41'20, trim_to=0'0, roll_forward_to=40'16))", 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 41.774410643000003, 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013627653999999999, 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.721 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920831+0000", 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.722 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920833+0000", 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.809e-06 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920890+0000", 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6799000000000003e-05 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920921+0000", 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1161999999999997e-05 2024-09-16T04:02:08.723 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920944+0000", 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3198999999999999e-05 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.926110+0000", 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051653740000000004 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.724 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.926136+0000", 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6568999999999999e-05 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.934366+0000", 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0082292779999999996 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.934459+0000", 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.3463999999999998e-05 2024-09-16T04:02:08.725 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 75/33 ECSubWrite(tid=282, reqid=client.4646.0:706, at_version=75'166, trim_to=0'0, roll_forward_to=73'164))", 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:01.036561+0000", 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 7.658681101, 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014939829, 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.726 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.036561+0000", 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.036561+0000", 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.036562+0000", 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6899999999999996e-07 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.727 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.038474+0000", 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001912422 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.038491+0000", 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6606999999999999e-05 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.038502+0000", 2024-09-16T04:02:08.728 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1017e-05 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.038663+0000", 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016080200000000001 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.038690+0000", 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6903e-05 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.729 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.051463+0000", 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012773652999999999 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.051501+0000", 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7555999999999998e-05 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 76/33 ECSubWrite(tid=284, reqid=client.4646.0:719, at_version=76'168, trim_to=0'0, roll_forward_to=75'166))", 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:01.748409+0000", 2024-09-16T04:02:08.730 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 6.9468335119999995, 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012173948, 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.748409+0000", 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.731 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.748409+0000", 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.748409+0000", 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7300000000000002e-07 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.748645+0000", 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000236491 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.732 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.748675+0000", 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9592999999999999e-05 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.748692+0000", 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7122999999999999e-05 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.733 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.748906+0000", 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000214386 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.748930+0000", 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3739000000000001e-05 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.760516+0000", 2024-09-16T04:02:08.734 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011585998 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.760582+0000", 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6245e-05 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 77/33 ECSubWrite(tid=309, reqid=osd.11.0:309, at_version=77'191, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:08.735 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:02.738906+0000", 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.9563356870000002, 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011870667999999999, 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.738906+0000", 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.736 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.738906+0000", 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.738908+0000", 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1799999999999999e-06 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.738958+0000", 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0803000000000001e-05 2024-09-16T04:02:08.737 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739233+0000", 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00027467099999999999 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739255+0000", 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2425999999999998e-05 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.738 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.740788+0000", 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0015321480000000001 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.740816+0000", 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8520000000000001e-05 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.750701+0000", 2024-09-16T04:02:08.739 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0098847919999999999 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.750777+0000", 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6128e-05 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.740 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 77/33 ECSubWrite(tid=53, reqid=osd.15.0:53, at_version=77'43, trim_to=0'0, roll_forward_to=73'41))", 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:02.739558+0000", 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.955684346, 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013835228999999999, 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739558+0000", 2024-09-16T04:02:08.741 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739558+0000", 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739558+0000", 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.742 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739612+0000", 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3814999999999999e-05 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739644+0000", 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1983000000000002e-05 2024-09-16T04:02:08.743 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739667+0000", 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2999000000000001e-05 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739831+0000", 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016352000000000001 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.744 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.739854+0000", 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3383e-05 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.753314+0000", 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013460021000000001 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.753393+0000", 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8857999999999996e-05 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.745 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 77/33 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=77'45, trim_to=0'0, roll_forward_to=73'41))", 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:02.744490+0000", 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.9507521590000003, 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013619308, 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744490+0000", 2024-09-16T04:02:08.746 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744490+0000", 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744492+0000", 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.72e-06 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.747 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744583+0000", 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.1181999999999995e-05 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744628+0000", 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4776e-05 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744659+0000", 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1724999999999999e-05 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.748 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744834+0000", 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017469999999999999 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744870+0000", 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5992999999999999e-05 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.757922+0000", 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013052339 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.749 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.758109+0000", 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000186873 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.es5 77/33 ECSubWrite(tid=27, reqid=osd.2.0:27, at_version=77'10, trim_to=0'0, roll_forward_to=73'8))", 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:02.744774+0000", 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 5.9504681179999999, 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013270872, 2024-09-16T04:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744774+0000", 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744774+0000", 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.751 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744774+0000", 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3000000000000001e-07 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744823+0000", 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8176000000000001e-05 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744847+0000", 2024-09-16T04:02:08.752 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3946000000000001e-05 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.744865+0000", 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8845999999999999e-05 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.745024+0000", 2024-09-16T04:02:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015822699999999999 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.745048+0000", 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4499e-05 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.757963+0000", 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0129146 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.758045+0000", 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2047999999999998e-05 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.bs4 82/33 ECSubWrite(tid=234, reqid=osd.4.0:234, at_version=82'127, trim_to=0'0, roll_forward_to=81'123))", 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.777118+0000", 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.91812428400000001, 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0128483, 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.755 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.777118+0000", 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.777118+0000", 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.756 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.777119+0000", 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.066e-06 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.777173+0000", 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3783000000000003e-05 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.777195+0000", 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2446000000000002e-05 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.757 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.777212+0000", 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6492999999999999e-05 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.781489+0000", 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042770960000000002 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.781515+0000", 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6415999999999999e-05 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.758 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.789899+0000", 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0083841850000000006 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.789966+0000", 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6815000000000003e-05 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.759 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 82/33 ECSubWrite(tid=166, reqid=osd.14.0:166, at_version=82'108, trim_to=0'0, roll_forward_to=81'104))", 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.785418+0000", 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.90982378699999999, 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011428667999999999, 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.785418+0000", 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.760 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.785418+0000", 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.785420+0000", 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.793e-06 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.785475+0000", 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5115999999999999e-05 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.761 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.762 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.762 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.785503+0000", 2024-09-16T04:02:08.762 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7844999999999999e-05 2024-09-16T04:02:08.762 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.762 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.762 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.762 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.785523+0000", 2024-09-16T04:02:08.762 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0486999999999999e-05 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.785695+0000", 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000171137 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.785718+0000", 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2895999999999999e-05 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.763 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796735+0000", 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011017476999999999 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796847+0000", 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00011191700000000001 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.764 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.5s6 82/33 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=82'53, trim_to=0'0, roll_forward_to=81'49))", 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.787069+0000", 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.90817352299999998, 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011738419, 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.765 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.787069+0000", 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.787069+0000", 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.787075+0000", 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.5230000000000001e-06 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.766 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.787126+0000", 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0939000000000001e-05 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.787157+0000", 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0576e-05 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.787179+0000", 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2436e-05 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.767 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.789802+0000", 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0026231380000000001 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.789828+0000", 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.597e-05 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798762+0000", 2024-09-16T04:02:08.768 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0089340530000000008 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798807+0000", 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4783999999999997e-05 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs4 82/33 ECSubWrite(tid=354, reqid=osd.11.0:354, at_version=82'231, trim_to=0'0, roll_forward_to=81'229))", 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.796021+0000", 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.89922107699999998, 2024-09-16T04:02:08.769 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0109885, 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796021+0000", 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796021+0000", 2024-09-16T04:02:08.770 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796022+0000", 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3e-07 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796079+0000", 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7156e-05 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.771 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796123+0000", 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.4186e-05 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796149+0000", 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6392000000000001e-05 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796355+0000", 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00020550599999999999 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.772 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796417+0000", 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.2545999999999994e-05 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.806908+0000", 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010490983000000001 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.807009+0000", 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000101001 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.773 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.774 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.774 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.774 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.774 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s6 82/33 ECSubWrite(tid=31, reqid=osd.5.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=82'25))", 2024-09-16T04:02:08.774 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.796346+0000", 2024-09-16T04:02:08.774 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.89889601699999999, 2024-09-16T04:02:08.774 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010549902, 2024-09-16T04:02:08.774 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796346+0000", 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.775 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796346+0000", 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796347+0000", 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4169999999999999e-06 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796364+0000", 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6101999999999999e-05 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.776 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796383+0000", 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9558000000000001e-05 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796398+0000", 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4799e-05 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796519+0000", 2024-09-16T04:02:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012107700000000001 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796532+0000", 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3388e-05 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.806835+0000", 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010302604 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.778 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.806896+0000", 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.0956999999999999e-05 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.as5 82/33 ECSubWrite(tid=31, reqid=osd.3.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=82'16))", 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.798298+0000", 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.89694365799999998, 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012491044, 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.779 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798298+0000", 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798298+0000", 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.780 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798299+0000", 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.1999999999999998e-07 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798320+0000", 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0582999999999999e-05 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798343+0000", 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3136e-05 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.781 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798365+0000", 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1566e-05 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798728+0000", 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00036302600000000002 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.798749+0000", 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1047e-05 2024-09-16T04:02:08.782 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.810707+0000", 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011958607 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.810789+0000", 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2158999999999997e-05 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.783 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s5 82/33 ECSubWrite(tid=169, reqid=osd.14.0:169, at_version=82'110, trim_to=0'0, roll_forward_to=82'108))", 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.801970+0000", 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.89327204599999999, 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013105598, 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.784 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801970+0000", 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801970+0000", 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.785 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801971+0000", 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.24e-06 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801998+0000", 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7064e-05 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.786 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.802014+0000", 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6095000000000001e-05 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.802025+0000", 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0939e-05 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.787 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.802160+0000", 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000134284 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.802183+0000", 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3782999999999999e-05 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.815029+0000", 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012845505 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.788 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.815076+0000", 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.6687999999999998e-05 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.bs4 82/33 ECSubWrite(tid=241, reqid=osd.4.0:241, at_version=82'129, trim_to=0'0, roll_forward_to=82'127))", 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.803748+0000", 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.891493969, 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011410177000000001, 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.789 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.803748+0000", 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.803748+0000", 2024-09-16T04:02:08.790 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.803750+0000", 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1500000000000002e-06 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.803825+0000", 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4375000000000005e-05 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.791 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.803859+0000", 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4542999999999998e-05 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.803884+0000", 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4433000000000001e-05 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.803981+0000", 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.7612e-05 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.792 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.803992+0000", 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0614e-05 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.815073+0000", 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.01108144 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.815158+0000", 2024-09-16T04:02:08.793 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.5010000000000001e-05 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.bs4 82/33 ECSubWrite(tid=242, reqid=osd.4.0:242, at_version=82'131, trim_to=0'0, roll_forward_to=82'127))", 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.807926+0000", 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 0.88731559699999996, 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010923865, 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.807926+0000", 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.807926+0000", 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:08.795 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.807927+0000", 2024-09-16T04:02:08.796 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0419999999999999e-06 2024-09-16T04:02:08.797 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.797 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.797 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:08.797 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.807952+0000", 2024-09-16T04:02:08.797 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4817e-05 2024-09-16T04:02:08.797 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.797 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.797 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:08.798 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.807965+0000", 2024-09-16T04:02:08.798 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3019e-05 2024-09-16T04:02:08.798 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.798 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.798 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:08.798 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.807975+0000", 2024-09-16T04:02:08.798 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.4900000000000006e-06 2024-09-16T04:02:08.799 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.799 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.799 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:08.799 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.808110+0000", 2024-09-16T04:02:08.799 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00013488700000000001 2024-09-16T04:02:08.799 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.799 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.799 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.808135+0000", 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5689000000000001e-05 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.818803+0000", 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010667483 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.818850+0000", 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7438000000000002e-05 2024-09-16T04:02:08.800 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.801 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.801 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.801 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:08.801 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:08.801 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:08.803 INFO:tasks.rados.rados.0.smithi188.stdout:319: done (2 left) 2024-09-16T04:02:08.803 INFO:tasks.rados.rados.0.smithi188.stdout:320: done (1 left) 2024-09-16T04:02:08.803 INFO:tasks.rados.rados.0.smithi188.stdout:322: done (0 left) 2024-09-16T04:02:08.803 INFO:tasks.rados.rados.0.smithi188.stdout:323: read oid 31 snap -1 2024-09-16T04:02:08.803 INFO:tasks.rados.rados.0.smithi188.stdout:323: expect (ObjNum 97 snap 19 seq_num 97) 2024-09-16T04:02:08.803 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 17 current snap is 33 2024-09-16T04:02:08.803 INFO:tasks.rados.rados.0.smithi188.stdout:324: seq_num 133 ranges {2342912=688128,3031040=720896,3751936=475136,4227072=360448} 2024-09-16T04:02:08.803 INFO:tasks.rados.rados.0.smithi188.stdout:324: writing smithi18831615-17 from 2342912 to 3031040 tid 1 2024-09-16T04:02:08.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:02:08.804 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:08.800+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:08.805 INFO:tasks.rados.rados.0.smithi188.stdout:324: writing smithi18831615-17 from 3031040 to 3751936 tid 2 2024-09-16T04:02:08.809 INFO:tasks.rados.rados.0.smithi188.stdout:324: writing smithi18831615-17 from 3751936 to 4227072 tid 3 2024-09-16T04:02:08.813 INFO:tasks.rados.rados.0.smithi188.stdout:324: writing smithi18831615-17 from 4227072 to 4587520 tid 4 2024-09-16T04:02:08.813 INFO:tasks.rados.rados.0.smithi188.stdout:325: snap_remove snap 26 2024-09-16T04:02:08.859 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:08.859 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:08.880 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:08.880 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:08.900 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:08.900 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:08.902 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:08.900+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:08.921 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:08.921 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:08.933 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:08.933 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:08.933 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:02:08.933 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:08.938 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:08.938 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:08.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:02:08.960 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:08.961 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:08.979 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:08.980 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:09.000 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:09.001 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:09.003 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:08.997+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:09.021 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:09.021 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:09.043 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:09.043 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:09.067 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:09.070 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:09.070 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:09.085 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:02:09.092 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:09.093 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:09.113 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:09.113 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:09.128 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:09.125+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:09.137 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:09.137 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:09.158 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:09.158 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:09.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:02:09.156+0000 7f23b6d9d640 0 -- 172.21.15.37:0/1389085912 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7f23b8225df0 msgr2=0x7f23b82281e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:02:09.205 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:09.201+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:09.251 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:09.251 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:09.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:02:09.309 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:02:09.305+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:02:09.387 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:09.407 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:09.404+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:09.433 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:09.434 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:09.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:02:09.507 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:09.501+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:09.571 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:09.571 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:09.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:02:09.608 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:09.604+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:09.709 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:09.705+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:09.756 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:09.756 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:09.771 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:02:09.795 INFO:tasks.rados.rados.0.smithi188.stdout:325: done (2 left) 2024-09-16T04:02:09.795 INFO:tasks.rados.rados.0.smithi188.stdout:326: snap_create 2024-09-16T04:02:09.795 INFO:tasks.rados.rados.0.smithi188.stdout:323: expect (ObjNum 97 snap 19 seq_num 97) 2024-09-16T04:02:09.806 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:02:09.851 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:09.848+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: finishing write tid 1 to smithi18831615-17 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: finishing write tid 2 to smithi18831615-17 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: finishing write tid 3 to smithi18831615-17 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: finishing write tid 4 to smithi18831615-17 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: finishing write tid 5 to smithi18831615-17 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: finishing write tid 6 to smithi18831615-17 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: oid 17 updating version 0 to 40 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: oid 17 version 40 is already newer than 39 2024-09-16T04:02:09.853 INFO:tasks.rados.rados.0.smithi188.stdout:324: oid 17 version 40 is already newer than 37 2024-09-16T04:02:09.854 INFO:tasks.rados.rados.0.smithi188.stdout:324: oid 17 version 40 is already newer than 38 2024-09-16T04:02:09.854 INFO:tasks.rados.rados.0.smithi188.stdout:324: oid 17 version 40 is already newer than 36 2024-09-16T04:02:09.854 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 17 v 40 (ObjNum 133 snap 33 seq_num 133) dirty exists 2024-09-16T04:02:09.854 INFO:tasks.rados.rados.0.smithi188.stdout:324: left oid 17 (ObjNum 133 snap 33 seq_num 133) 2024-09-16T04:02:09.911 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:09.908+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:09.917 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:09.918 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:09.931 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:02:10.012 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:02:10.009+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:02:10.098 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:10.099 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:10.113 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:02:10.114 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:10.109+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:10.214 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:10.212+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:10.243 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:10.243 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:10.257 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:02:10.314 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:10.312+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:10.381 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:10.717 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:10.719 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:02:10.719 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:10.716+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-09-16T04:02:10.810 INFO:tasks.rados.rados.0.smithi188.stdout:323: done (2 left) 2024-09-16T04:02:10.810 INFO:tasks.rados.rados.0.smithi188.stdout:324: done (1 left) 2024-09-16T04:02:10.810 INFO:tasks.rados.rados.0.smithi188.stdout:326: done (0 left) 2024-09-16T04:02:10.810 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 34 current snap is 34 2024-09-16T04:02:10.810 INFO:tasks.rados.rados.0.smithi188.stdout:327: seq_num 134 ranges {720896=704512,1425408=475136,1900544=573440,2473984=507904} 2024-09-16T04:02:10.820 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:10.816+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:10.827 INFO:tasks.rados.rados.0.smithi188.stdout:327: writing smithi18831615-34 from 720896 to 1425408 tid 1 2024-09-16T04:02:10.833 INFO:tasks.rados.rados.0.smithi188.stdout:327: writing smithi18831615-34 from 1425408 to 1900544 tid 2 2024-09-16T04:02:10.841 INFO:tasks.rados.rados.0.smithi188.stdout:327: writing smithi18831615-34 from 1900544 to 2473984 tid 3 2024-09-16T04:02:10.849 INFO:tasks.rados.rados.0.smithi188.stdout:327: writing smithi18831615-34 from 2473984 to 2981888 tid 4 2024-09-16T04:02:10.849 INFO:tasks.rados.rados.0.smithi188.stdout:328: rollback oid 42 current snap is 34 2024-09-16T04:02:10.849 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 42 to 8 2024-09-16T04:02:10.849 INFO:tasks.rados.rados.0.smithi188.stdout:327: finishing write tid 1 to smithi18831615-34 2024-09-16T04:02:10.850 INFO:tasks.rados.rados.0.smithi188.stdout:327: finishing write tid 2 to smithi18831615-34 2024-09-16T04:02:10.850 INFO:tasks.rados.rados.0.smithi188.stdout:329: delete oid 40 current snap is 34 2024-09-16T04:02:10.850 INFO:tasks.rados.rados.0.smithi188.stdout:327: finishing write tid 3 to smithi18831615-34 2024-09-16T04:02:10.857 INFO:tasks.rados.rados.0.smithi188.stdout:327: finishing write tid 4 to smithi18831615-34 2024-09-16T04:02:10.857 INFO:tasks.rados.rados.0.smithi188.stdout:328: finishing rollback tid 1 to smithi18831615-42 2024-09-16T04:02:10.858 INFO:tasks.rados.rados.0.smithi188.stdout:329: done (2 left) 2024-09-16T04:02:10.858 INFO:tasks.rados.rados.0.smithi188.stdout:330: rmattr oid 27 current snap is 34 2024-09-16T04:02:10.858 INFO:tasks.rados.rados.0.smithi188.stdout:330: done (2 left) 2024-09-16T04:02:10.859 INFO:tasks.rados.rados.0.smithi188.stdout:331: setattr oid 13 current snap is 34 2024-09-16T04:02:10.859 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:10.859 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:10.862 INFO:tasks.rados.rados.0.smithi188.stdout:327: finishing write tid 5 to smithi18831615-34 2024-09-16T04:02:10.862 INFO:tasks.rados.rados.0.smithi188.stdout:328: finishing rollback tid 2 to smithi18831615-42 2024-09-16T04:02:10.862 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 42 v 20 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T04:02:10.862 INFO:tasks.rados.rados.0.smithi188.stdout:328: done (2 left) 2024-09-16T04:02:10.862 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 46 current snap is 34 2024-09-16T04:02:10.862 INFO:tasks.rados.rados.0.smithi188.stdout:332: seq_num 136 ranges {0=737280,737280=770048,1507328=540672,2048000=245760} 2024-09-16T04:02:10.866 INFO:tasks.rados.rados.0.smithi188.stdout:332: writing smithi18831615-46 from 0 to 737280 tid 1 2024-09-16T04:02:10.871 INFO:tasks.rados.rados.0.smithi188.stdout:332: writing smithi18831615-46 from 737280 to 1507328 tid 2 2024-09-16T04:02:10.876 INFO:tasks.rados.rados.0.smithi188.stdout:332: writing smithi18831615-46 from 1507328 to 2048000 tid 3 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:332: writing smithi18831615-46 from 2048000 to 2293760 tid 4 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:333: read oid 10 snap -1 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:333: expect (ObjNum 131 snap 32 seq_num 131) 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:334: setattr oid 3 current snap is 34 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:327: finishing write tid 6 to smithi18831615-34 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:327: oid 34 updating version 0 to 91 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:327: oid 34 version 91 is already newer than 90 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:327: oid 34 version 91 is already newer than 87 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:327: oid 34 version 91 is already newer than 88 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:327: oid 34 version 91 is already newer than 89 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 34 v 91 (ObjNum 134 snap 34 seq_num 134) dirty exists 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:327: left oid 34 (ObjNum 134 snap 34 seq_num 134) 2024-09-16T04:02:10.877 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 13 v 79 (ObjNum 129 snap 0 seq_num 3906002616) dirty exists 2024-09-16T04:02:10.879 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:02:10.879 INFO:tasks.rados.rados.0.smithi188.stdout:332: finishing write tid 1 to smithi18831615-46 2024-09-16T04:02:10.880 INFO:tasks.rados.rados.0.smithi188.stdout:327: done (4 left) 2024-09-16T04:02:10.880 INFO:tasks.rados.rados.0.smithi188.stdout:331: done (3 left) 2024-09-16T04:02:10.880 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 41 current snap is 34 2024-09-16T04:02:10.880 INFO:tasks.rados.rados.0.smithi188.stdout:335: seq_num 137 ranges {688128=753664,1441792=786432,2228224=655360,2883584=114688} 2024-09-16T04:02:10.889 INFO:tasks.rados.rados.0.smithi188.stdout:335: writing smithi18831615-41 from 688128 to 1441792 tid 1 2024-09-16T04:02:10.895 INFO:tasks.rados.rados.0.smithi188.stdout:335: writing smithi18831615-41 from 1441792 to 2228224 tid 2 2024-09-16T04:02:10.900 INFO:tasks.rados.rados.0.smithi188.stdout:335: writing smithi18831615-41 from 2228224 to 2883584 tid 3 2024-09-16T04:02:10.901 INFO:tasks.rados.rados.0.smithi188.stdout:335: writing smithi18831615-41 from 2883584 to 2998272 tid 4 2024-09-16T04:02:10.901 INFO:tasks.rados.rados.0.smithi188.stdout:336: copy_from oid 39 from oid 26 current snap is 34 2024-09-16T04:02:10.901 INFO:tasks.rados.rados.0.smithi188.stdout:332: finishing write tid 2 to smithi18831615-46 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: finishing write tid 3 to smithi18831615-46 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: finishing write tid 4 to smithi18831615-46 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: finishing write tid 5 to smithi18831615-46 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: finishing write tid 6 to smithi18831615-46 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: oid 46 updating version 0 to 128 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: oid 46 version 128 is already newer than 125 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: oid 46 version 128 is already newer than 126 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: oid 46 version 128 is already newer than 127 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: oid 46 updating version 128 to 129 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 46 v 129 (ObjNum 136 snap 34 seq_num 136) dirty exists 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:332: left oid 46 (ObjNum 136 snap 34 seq_num 136) 2024-09-16T04:02:10.902 INFO:tasks.rados.rados.0.smithi188.stdout:333: expect (ObjNum 131 snap 32 seq_num 131) 2024-09-16T04:02:10.921 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:10.916+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:10.985 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 3 v 130 (ObjNum 209 snap 0 seq_num 3905950832) dirty exists 2024-09-16T04:02:10.985 INFO:tasks.rados.rados.0.smithi188.stdout:335: finishing write tid 1 to smithi18831615-41 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: finishing write tid 2 to smithi18831615-41 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: finishing write tid 3 to smithi18831615-41 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: finishing write tid 4 to smithi18831615-41 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:332: done (4 left) 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:333: done (3 left) 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:334: done (2 left) 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 32 current snap is 34 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:336: finishing copy_from racing read to smithi18831615-39 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: finishing write tid 5 to smithi18831615-41 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: finishing write tid 6 to smithi18831615-41 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: oid 41 updating version 0 to 31 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: oid 41 version 31 is already newer than 30 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: oid 41 updating version 31 to 32 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: oid 41 updating version 32 to 33 2024-09-16T04:02:10.986 INFO:tasks.rados.rados.0.smithi188.stdout:335: oid 41 version 33 is already newer than 29 2024-09-16T04:02:10.987 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 41 v 33 (ObjNum 137 snap 34 seq_num 137) dirty exists 2024-09-16T04:02:10.987 INFO:tasks.rados.rados.0.smithi188.stdout:335: left oid 41 (ObjNum 137 snap 34 seq_num 137) 2024-09-16T04:02:10.987 INFO:tasks.rados.rados.0.smithi188.stdout:336: finishing copy_from to smithi18831615-39 2024-09-16T04:02:10.987 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 39 v 2 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T04:02:10.987 INFO:tasks.rados.rados.0.smithi188.stdout:337: seq_num 138 ranges {2129920=770048,2899968=540672,3440640=753664,4194304=262144} 2024-09-16T04:02:11.001 INFO:tasks.rados.rados.0.smithi188.stdout:337: writing smithi18831615-32 from 2129920 to 2899968 tid 1 2024-09-16T04:02:11.001 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:11.002 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:11.004 INFO:tasks.rados.rados.0.smithi188.stdout:337: writing smithi18831615-32 from 2899968 to 3440640 tid 2 2024-09-16T04:02:11.008 INFO:tasks.rados.rados.0.smithi188.stdout:337: writing smithi18831615-32 from 3440640 to 4194304 tid 3 2024-09-16T04:02:11.012 INFO:tasks.rados.rados.0.smithi188.stdout:337: writing smithi18831615-32 from 4194304 to 4456448 tid 4 2024-09-16T04:02:11.012 INFO:tasks.rados.rados.0.smithi188.stdout:335: done (2 left) 2024-09-16T04:02:11.012 INFO:tasks.rados.rados.0.smithi188.stdout:336: done (1 left) 2024-09-16T04:02:11.012 INFO:tasks.rados.rados.0.smithi188.stdout:338: read oid 41 snap -1 2024-09-16T04:02:11.012 INFO:tasks.rados.rados.0.smithi188.stdout:338: expect (ObjNum 137 snap 34 seq_num 137) 2024-09-16T04:02:11.012 INFO:tasks.rados.rados.0.smithi188.stdout:339: snap_remove snap 32 2024-09-16T04:02:11.017 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:02:11.059 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:11.056+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:11.123 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:11.120+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:11.178 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:11.178 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:11.191 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:02:11.224 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:11.220+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:11.325 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:11.321+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:11.352 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:11.352 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:11.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:02:11.413 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:02:11.426 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:11.420+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:11.487 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:11.488 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:11.501 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:02:11.527 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:11.524+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:11.570 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:11.570 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:11.591 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:11.591 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:11.611 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:11.611 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:11.628 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:11.624+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:11.630 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:11.630 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:11.649 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:11.650 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:11.667 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:11.668 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:11.672 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:11.682 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:02:11.695 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:11.696 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:11.717 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:11.717 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:11.728 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:11.724+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:11.733 INFO:tasks.rados.rados.0.smithi188.stdout:339: done (2 left) 2024-09-16T04:02:11.733 INFO:tasks.rados.rados.0.smithi188.stdout:340: read oid 36 snap -1 2024-09-16T04:02:11.733 INFO:tasks.rados.rados.0.smithi188.stdout:340: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:02:11.733 INFO:tasks.rados.rados.0.smithi188.stdout:341: delete oid 26 current snap is 34 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: finishing write tid 1 to smithi18831615-32 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: finishing write tid 2 to smithi18831615-32 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: finishing write tid 3 to smithi18831615-32 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: finishing write tid 4 to smithi18831615-32 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: finishing write tid 5 to smithi18831615-32 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: finishing write tid 6 to smithi18831615-32 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: oid 32 updating version 0 to 45 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: oid 32 updating version 45 to 46 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: oid 32 version 46 is already newer than 44 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: oid 32 updating version 46 to 47 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: oid 32 version 47 is already newer than 43 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 32 v 47 (ObjNum 138 snap 34 seq_num 138) dirty exists 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:337: left oid 32 (ObjNum 138 snap 34 seq_num 138) 2024-09-16T04:02:11.734 INFO:tasks.rados.rados.0.smithi188.stdout:338: expect (ObjNum 137 snap 34 seq_num 137) 2024-09-16T04:02:11.739 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:11.740 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:11.759 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:02:11.759 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:02:11.759 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-16T04:02:11.760 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 16 2024-09-16T04:02:11.760 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:02:11.814 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:11.814 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:11.830 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:02:11.830 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:11.828+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:11.833 INFO:tasks.rados.rados.0.smithi188.stdout:340: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:02:11.845 INFO:tasks.rados.rados.0.smithi188.stdout:337: done (3 left) 2024-09-16T04:02:11.845 INFO:tasks.rados.rados.0.smithi188.stdout:338: done (2 left) 2024-09-16T04:02:11.845 INFO:tasks.rados.rados.0.smithi188.stdout:340: done (1 left) 2024-09-16T04:02:11.845 INFO:tasks.rados.rados.0.smithi188.stdout:341: done (0 left) 2024-09-16T04:02:11.845 INFO:tasks.rados.rados.0.smithi188.stdout:342: read oid 14 snap 30 2024-09-16T04:02:11.845 INFO:tasks.rados.rados.0.smithi188.stdout:342: expect deleted 2024-09-16T04:02:11.845 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 42 current snap is 34 2024-09-16T04:02:11.845 INFO:tasks.rados.rados.0.smithi188.stdout:343: seq_num 140 ranges {704512=802816,1507328=425984,1933312=720896,2654208=409600} 2024-09-16T04:02:11.854 INFO:tasks.rados.rados.0.smithi188.stdout:343: writing smithi18831615-42 from 704512 to 1507328 tid 1 2024-09-16T04:02:11.858 INFO:tasks.rados.rados.0.smithi188.stdout:343: writing smithi18831615-42 from 1507328 to 1933312 tid 2 2024-09-16T04:02:11.864 INFO:tasks.rados.rados.0.smithi188.stdout:343: writing smithi18831615-42 from 1933312 to 2654208 tid 3 2024-09-16T04:02:11.867 INFO:tasks.rados.rados.0.smithi188.stdout:343: writing smithi18831615-42 from 2654208 to 3063808 tid 4 2024-09-16T04:02:11.867 INFO:tasks.rados.rados.0.smithi188.stdout:344: read oid 32 snap -1 2024-09-16T04:02:11.867 INFO:tasks.rados.rados.0.smithi188.stdout:344: expect (ObjNum 138 snap 34 seq_num 138) 2024-09-16T04:02:11.867 INFO:tasks.rados.rados.0.smithi188.stdout:345: snap_create 2024-09-16T04:02:11.868 INFO:tasks.rados.rados.0.smithi188.stdout:343: finishing write tid 1 to smithi18831615-42 2024-09-16T04:02:11.868 INFO:tasks.rados.rados.0.smithi188.stdout:343: finishing write tid 2 to smithi18831615-42 2024-09-16T04:02:11.871 INFO:tasks.rados.rados.0.smithi188.stdout:343: finishing write tid 3 to smithi18831615-42 2024-09-16T04:02:11.873 INFO:tasks.rados.rados.0.smithi188.stdout:343: finishing write tid 4 to smithi18831615-42 2024-09-16T04:02:11.874 INFO:tasks.rados.rados.0.smithi188.stdout:343: finishing write tid 5 to smithi18831615-42 2024-09-16T04:02:11.877 INFO:tasks.rados.rados.0.smithi188.stdout:343: finishing write tid 6 to smithi18831615-42 2024-09-16T04:02:11.877 INFO:tasks.rados.rados.0.smithi188.stdout:343: oid 42 updating version 0 to 22 2024-09-16T04:02:11.877 INFO:tasks.rados.rados.0.smithi188.stdout:343: oid 42 updating version 22 to 23 2024-09-16T04:02:11.877 INFO:tasks.rados.rados.0.smithi188.stdout:343: oid 42 updating version 23 to 24 2024-09-16T04:02:11.877 INFO:tasks.rados.rados.0.smithi188.stdout:343: oid 42 version 24 is already newer than 21 2024-09-16T04:02:11.877 INFO:tasks.rados.rados.0.smithi188.stdout:343: oid 42 updating version 24 to 25 2024-09-16T04:02:11.877 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 42 v 25 (ObjNum 140 snap 34 seq_num 140) dirty exists 2024-09-16T04:02:11.877 INFO:tasks.rados.rados.0.smithi188.stdout:343: left oid 42 (ObjNum 140 snap 34 seq_num 140) 2024-09-16T04:02:11.903 INFO:tasks.rados.rados.0.smithi188.stdout:344: expect (ObjNum 138 snap 34 seq_num 138) 2024-09-16T04:02:11.930 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:11.924+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:11.965 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:11.965 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:11.969 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:11.970 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:11.979 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:02:11.993 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:11.994 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:12.023 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:12.023 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:12.030 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:12.028+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:12.040 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:12.040 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:12.061 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:12.061 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:12.074 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:02:12.074 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:02:12.084 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:12.084 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:12.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:02:12.080+0000 7f033903d640 0 -- 172.21.15.37:0/570623157 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7f03342266d0 msgr2=0x7f0334228ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:02:12.092 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-16T04:02:11.728131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":143983155,"num_objects":134,"num_object_clones":80,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":9,"num_read":555,"num_read_kb":188577,"num_write":986,"num_write_kb":148342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1176,"ondisk_log_size":1176,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":736384,"kb_used_data":253764,"kb_used_omap":24,"kb_used_meta":482023,"kb_avail":1509213056,"statfs":{"total":1546188226560,"available":1545434169344,"internally_reserved":0,"allocated":259854336,"data_stored":394145186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":493591633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9371648,"num_objects":5,"num_object_clones":5,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":-1,"num_read":17,"num_read_kb":19254,"num_write":59,"num_write_kb":9152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.960567"},"pg_stats":[{"pgid":"3.f","version":"85'79","reported_seq":311,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.856397+0000","last_change":"2024-09-16T04:02:09.849476+0000","last_active":"2024-09-16T04:02:10.856397+0000","last_peered":"2024-09-16T04:02:10.856397+0000","last_clean":"2024-09-16T04:02:10.856397+0000","last_became_active":"2024-09-16T04:01:19.845422+0000","last_became_peered":"2024-09-16T04:01:19.845422+0000","last_unstale":"2024-09-16T04:02:10.856397+0000","last_undegraded":"2024-09-16T04:02:10.856397+0000","last_fullsized":"2024-09-16T04:02:10.856397+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:30.945984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026470220000000002,"stat_sum":{"num_bytes":10682368,"num_objects":10,"num_object_clones":5,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":28,"num_read_kb":6986,"num_write":71,"num_write_kb":10960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.e","version":"82'16","reported_seq":244,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:02:07.800838+0000","last_change":"2024-09-16T04:02:07.805297+0000","last_active":"2024-09-16T04:02:07.800838+0000","last_peered":"2024-09-16T04:02:07.800838+0000","last_clean":"2024-09-16T04:02:07.800838+0000","last_became_active":"2024-09-16T04:01:20.266249+0000","last_became_peered":"2024-09-16T04:01:20.266249+0000","last_unstale":"2024-09-16T04:02:07.800838+0000","last_undegraded":"2024-09-16T04:02:07.800838+0000","last_fullsized":"2024-09-16T04:02:07.800838+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:44.148680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032005314999999999,"stat_sum":{"num_bytes":2703360,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":8118,"num_write":27,"num_write_kb":3040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.d","version":"81'71","reported_seq":290,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.775820+0000","last_change":"2024-09-16T04:02:06.779271+0000","last_active":"2024-09-16T04:02:06.775820+0000","last_peered":"2024-09-16T04:02:06.775820+0000","last_clean":"2024-09-16T04:02:06.775820+0000","last_became_active":"2024-09-16T04:01:19.845009+0000","last_became_peered":"2024-09-16T04:01:19.845009+0000","last_unstale":"2024-09-16T04:02:06.775820+0000","last_undegraded":"2024-09-16T04:02:06.775820+0000","last_fullsized":"2024-09-16T04:02:06.775820+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:32.428797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029091697999999999,"stat_sum":{"num_bytes":18071552,"num_objects":12,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":37,"num_read_kb":15589,"num_write":54,"num_write_kb":15680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"3.c","version":"84'253","reported_seq":480,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.856246+0000","last_change":"2024-09-16T04:02:09.888387+0000","last_active":"2024-09-16T04:02:10.856246+0000","last_peered":"2024-09-16T04:02:10.856246+0000","last_clean":"2024-09-16T04:02:10.856246+0000","last_became_active":"2024-09-16T04:01:20.259789+0000","last_became_peered":"2024-09-16T04:01:20.259789+0000","last_unstale":"2024-09-16T04:02:10.856246+0000","last_undegraded":"2024-09-16T04:02:10.856246+0000","last_fullsized":"2024-09-16T04:02:10.856246+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:47.517272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.041930749000000003,"stat_sum":{"num_bytes":35848192,"num_objects":27,"num_object_clones":19,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":1,"num_read":63,"num_read_kb":21525,"num_write":172,"num_write_kb":36208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.b","version":"85'141","reported_seq":387,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.916828+0000","last_change":"2024-09-16T04:02:09.865787+0000","last_active":"2024-09-16T04:02:10.916828+0000","last_peered":"2024-09-16T04:02:10.916828+0000","last_clean":"2024-09-16T04:02:10.916828+0000","last_became_active":"2024-09-16T04:01:20.268548+0000","last_became_peered":"2024-09-16T04:01:20.268548+0000","last_unstale":"2024-09-16T04:02:10.916828+0000","last_undegraded":"2024-09-16T04:02:10.916828+0000","last_fullsized":"2024-09-16T04:02:10.916828+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.648043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026647919999999999,"stat_sum":{"num_bytes":21692416,"num_objects":17,"num_object_clones":11,"num_object_copies":119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":66,"num_read_kb":36935,"num_write":99,"num_write_kb":22096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"3.a","version":"81'14","reported_seq":230,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.774463+0000","last_change":"2024-09-16T04:02:06.777185+0000","last_active":"2024-09-16T04:02:06.774463+0000","last_peered":"2024-09-16T04:02:06.774463+0000","last_clean":"2024-09-16T04:02:06.774463+0000","last_became_active":"2024-09-16T04:01:20.268001+0000","last_became_peered":"2024-09-16T04:01:20.268001+0000","last_unstale":"2024-09-16T04:02:06.774463+0000","last_undegraded":"2024-09-16T04:02:06.774463+0000","last_fullsized":"2024-09-16T04:02:06.774463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:36.114360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021773317,"stat_sum":{"num_bytes":704512,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":19,"num_read_kb":8261,"num_write":5,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"3.9","version":"82'27","reported_seq":249,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:02:07.790715+0000","last_change":"2024-09-16T04:02:07.792068+0000","last_active":"2024-09-16T04:02:07.790715+0000","last_peered":"2024-09-16T04:02:07.790715+0000","last_clean":"2024-09-16T04:02:07.790715+0000","last_became_active":"2024-09-16T04:01:20.153728+0000","last_became_peered":"2024-09-16T04:01:20.153728+0000","last_unstale":"2024-09-16T04:02:07.790715+0000","last_undegraded":"2024-09-16T04:02:07.790715+0000","last_fullsized":"2024-09-16T04:02:07.790715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.765669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016813827999999999,"stat_sum":{"num_bytes":2441216,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7157,"num_write":10,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.8","version":"81'23","reported_seq":226,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.771092+0000","last_change":"2024-09-16T04:02:06.773465+0000","last_active":"2024-09-16T04:02:06.771092+0000","last_peered":"2024-09-16T04:02:06.771092+0000","last_clean":"2024-09-16T04:02:06.771092+0000","last_became_active":"2024-09-16T04:01:20.259482+0000","last_became_peered":"2024-09-16T04:01:20.259482+0000","last_unstale":"2024-09-16T04:02:06.771092+0000","last_undegraded":"2024-09-16T04:02:06.771092+0000","last_fullsized":"2024-09-16T04:02:06.771092+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:44.156090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025885073000000001,"stat_sum":{"num_bytes":3063808,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":11,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":318,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.738414+0000","last_change":"2024-09-16T04:00:59.019405+0000","last_active":"2024-09-16T04:02:06.738414+0000","last_peered":"2024-09-16T04:02:06.738414+0000","last_clean":"2024-09-16T04:02:06.738414+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:02:06.738414+0000","last_undegraded":"2024-09-16T04:02:06.738414+0000","last_fullsized":"2024-09-16T04:02:06.738414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:00:57.414587+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:08.815253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":118,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.747109+0000","last_change":"2024-09-16T04:01:08.987799+0000","last_active":"2024-09-16T04:02:06.747109+0000","last_peered":"2024-09-16T04:02:06.747109+0000","last_clean":"2024-09-16T04:02:06.747109+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:02:06.747109+0000","last_undegraded":"2024-09-16T04:02:06.747109+0000","last_fullsized":"2024-09-16T04:02:06.747109+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:10.746478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"79'34","reported_seq":253,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.739977+0000","last_change":"2024-09-16T04:02:06.740391+0000","last_active":"2024-09-16T04:02:06.739977+0000","last_peered":"2024-09-16T04:02:06.739977+0000","last_clean":"2024-09-16T04:02:06.739977+0000","last_became_active":"2024-09-16T04:01:20.258661+0000","last_became_peered":"2024-09-16T04:01:20.258661+0000","last_unstale":"2024-09-16T04:02:06.739977+0000","last_undegraded":"2024-09-16T04:02:06.739977+0000","last_fullsized":"2024-09-16T04:02:06.739977+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:46.276973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001183398,"stat_sum":{"num_bytes":6111232,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":20601,"num_write":24,"num_write_kb":6544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":119,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.753027+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:02:06.753027+0000","last_peered":"2024-09-16T04:02:06.753027+0000","last_clean":"2024-09-16T04:02:06.753027+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:02:06.753027+0000","last_undegraded":"2024-09-16T04:02:06.753027+0000","last_fullsized":"2024-09-16T04:02:06.753027+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:14.097441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"85'47","reported_seq":270,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:11.026628+0000","last_change":"2024-09-16T04:02:09.858354+0000","last_active":"2024-09-16T04:02:11.026628+0000","last_peered":"2024-09-16T04:02:11.026628+0000","last_clean":"2024-09-16T04:02:11.026628+0000","last_became_active":"2024-09-16T04:01:20.269891+0000","last_became_peered":"2024-09-16T04:01:20.269891+0000","last_unstale":"2024-09-16T04:02:11.026628+0000","last_undegraded":"2024-09-16T04:02:11.026628+0000","last_fullsized":"2024-09-16T04:02:11.026628+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:40.064239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019263491000000001,"stat_sum":{"num_bytes":7077888,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":1542,"num_write":58,"num_write_kb":6912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":121,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.738744+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:02:06.738744+0000","last_peered":"2024-09-16T04:02:06.738744+0000","last_clean":"2024-09-16T04:02:06.738744+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:02:06.738744+0000","last_undegraded":"2024-09-16T04:02:06.738744+0000","last_fullsized":"2024-09-16T04:02:06.738744+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"82'114","reported_seq":329,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:02:07.828990+0000","last_change":"2024-09-16T04:02:07.830199+0000","last_active":"2024-09-16T04:02:07.828990+0000","last_peered":"2024-09-16T04:02:07.828990+0000","last_clean":"2024-09-16T04:02:07.828990+0000","last_became_active":"2024-09-16T04:01:20.257526+0000","last_became_peered":"2024-09-16T04:01:20.257526+0000","last_unstale":"2024-09-16T04:02:07.828990+0000","last_undegraded":"2024-09-16T04:02:07.828990+0000","last_fullsized":"2024-09-16T04:02:07.828990+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:25.525688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055061911999999998,"stat_sum":{"num_bytes":8192000,"num_objects":14,"num_object_clones":9,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":4,"num_read":33,"num_read_kb":12352,"num_write":91,"num_write_kb":11696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"}]},{"pgid":"2.2","version":"30'2","reported_seq":121,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.754820+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:02:06.754820+0000","last_peered":"2024-09-16T04:02:06.754820+0000","last_clean":"2024-09-16T04:02:06.754820+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:02:06.754820+0000","last_undegraded":"2024-09-16T04:02:06.754820+0000","last_fullsized":"2024-09-16T04:02:06.754820+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:17.102095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"81'61","reported_seq":271,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.772486+0000","last_change":"2024-09-16T04:02:06.772486+0000","last_active":"2024-09-16T04:02:06.772486+0000","last_peered":"2024-09-16T04:02:06.772486+0000","last_clean":"2024-09-16T04:02:06.772486+0000","last_became_active":"2024-09-16T04:01:19.840099+0000","last_became_peered":"2024-09-16T04:01:19.840099+0000","last_unstale":"2024-09-16T04:02:06.772486+0000","last_undegraded":"2024-09-16T04:02:06.772486+0000","last_fullsized":"2024-09-16T04:02:06.772486+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:20.318332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033627754000000003,"stat_sum":{"num_bytes":9601024,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":18,"num_read_kb":12148,"num_write":62,"num_write_kb":7520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"3.5","version":"82'53","reported_seq":280,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.856547+0000","last_change":"2024-09-16T04:02:09.848135+0000","last_active":"2024-09-16T04:02:10.856547+0000","last_peered":"2024-09-16T04:02:10.856547+0000","last_clean":"2024-09-16T04:02:10.856547+0000","last_became_active":"2024-09-16T04:01:20.258954+0000","last_became_peered":"2024-09-16T04:01:20.258954+0000","last_unstale":"2024-09-16T04:02:10.856547+0000","last_undegraded":"2024-09-16T04:02:10.856547+0000","last_fullsized":"2024-09-16T04:02:10.856547+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.280960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070385800000000002,"stat_sum":{"num_bytes":7225344,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":4147,"num_write":38,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":126,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.802918+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:02:10.802918+0000","last_peered":"2024-09-16T04:02:10.802918+0000","last_clean":"2024-09-16T04:02:10.802918+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:02:10.802918+0000","last_undegraded":"2024-09-16T04:02:10.802918+0000","last_fullsized":"2024-09-16T04:02:10.802918+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"84'66","reported_seq":317,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.889133+0000","last_change":"2024-09-16T04:02:09.866454+0000","last_active":"2024-09-16T04:02:10.889133+0000","last_peered":"2024-09-16T04:02:10.889133+0000","last_clean":"2024-09-16T04:02:10.889133+0000","last_became_active":"2024-09-16T04:01:20.154175+0000","last_became_peered":"2024-09-16T04:01:20.154175+0000","last_unstale":"2024-09-16T04:02:10.889133+0000","last_undegraded":"2024-09-16T04:02:10.889133+0000","last_fullsized":"2024-09-16T04:02:10.889133+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:30.228524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025744783,"stat_sum":{"num_bytes":8273920,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":29872,"num_write":49,"num_write_kb":9440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":121,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:02:07.783093+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:02:07.783093+0000","last_peered":"2024-09-16T04:02:07.783093+0000","last_clean":"2024-09-16T04:02:07.783093+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:02:07.783093+0000","last_undegraded":"2024-09-16T04:02:07.783093+0000","last_fullsized":"2024-09-16T04:02:07.783093+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"84'15","reported_seq":243,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.803088+0000","last_change":"2024-09-16T04:02:09.871238+0000","last_active":"2024-09-16T04:02:10.803088+0000","last_peered":"2024-09-16T04:02:10.803088+0000","last_clean":"2024-09-16T04:02:10.803088+0000","last_became_active":"2024-09-16T04:01:20.266522+0000","last_became_peered":"2024-09-16T04:01:20.266522+0000","last_unstale":"2024-09-16T04:02:10.803088+0000","last_undegraded":"2024-09-16T04:02:10.803088+0000","last_fullsized":"2024-09-16T04:02:10.803088+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:22.139811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031744412,"stat_sum":{"num_bytes":1277952,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":724,"num_write":8,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"2"},{"start":"1c","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":127,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.837863+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:02:10.837863+0000","last_peered":"2024-09-16T04:02:10.837863+0000","last_clean":"2024-09-16T04:02:10.837863+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:02:10.837863+0000","last_undegraded":"2024-09-16T04:02:10.837863+0000","last_fullsized":"2024-09-16T04:02:10.837863+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"36'2","reported_seq":210,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-16T04:02:06.742398+0000","last_change":"2024-09-16T04:02:06.748450+0000","last_active":"2024-09-16T04:02:06.742398+0000","last_peered":"2024-09-16T04:02:06.742398+0000","last_clean":"2024-09-16T04:02:06.742398+0000","last_became_active":"2024-09-16T04:01:20.267489+0000","last_became_peered":"2024-09-16T04:01:20.267489+0000","last_unstale":"2024-09-16T04:02:06.742398+0000","last_undegraded":"2024-09-16T04:02:06.742398+0000","last_fullsized":"2024-09-16T04:02:06.742398+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:38.982150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063510650000000004,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2499,"num_write":2,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":127,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:02:10.856357+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:02:10.856357+0000","last_peered":"2024-09-16T04:02:10.856357+0000","last_clean":"2024-09-16T04:02:10.856357+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:02:10.856357+0000","last_undegraded":"2024-09-16T04:02:10.856357+0000","last_fullsized":"2024-09-16T04:02:10.856357+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":143392768,"num_objects":130,"num_object_clones":80,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":9,"num_read":419,"num_read_kb":188459,"num_write":781,"num_write_kb":144880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":247898112,"data_stored":394125312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1016,"ondisk_log_size":1016,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215122,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44596,"kb_used_data":13228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327244,"statfs":{"total":96636764160,"available":96591097856,"internally_reserved":0,"allocated":13545472,"data_stored":18580060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215122,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49628,"kb_used_data":18440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322212,"statfs":{"total":96636764160,"available":96585945088,"internally_reserved":0,"allocated":18882560,"data_stored":29885020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215123,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53420,"kb_used_data":22236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318420,"statfs":{"total":96636764160,"available":96582062080,"internally_reserved":0,"allocated":22769664,"data_stored":36778588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41740,"kb_used_data":14708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330100,"statfs":{"total":96636764160,"available":96594022400,"internally_reserved":0,"allocated":15060992,"data_stored":21791608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280532,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51944,"kb_used_data":20752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319896,"statfs":{"total":96636764160,"available":96583573504,"internally_reserved":0,"allocated":21250048,"data_stored":29961177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280530,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49100,"kb_used_data":22068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322740,"statfs":{"total":96636764160,"available":96586485760,"internally_reserved":0,"allocated":22597632,"data_stored":35246968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280530,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35808,"kb_used_data":8776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336032,"statfs":{"total":96636764160,"available":96600096768,"internally_reserved":0,"allocated":8986624,"data_stored":12084088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280530,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43932,"kb_used_data":16896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327908,"statfs":{"total":96636764160,"available":96591777792,"internally_reserved":0,"allocated":17301504,"data_stored":27397081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43044,"kb_used_data":16008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328796,"statfs":{"total":96636764160,"available":96592687104,"internally_reserved":0,"allocated":16392192,"data_stored":27565036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37340,"kb_used_data":10308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334500,"statfs":{"total":96636764160,"available":96598528000,"internally_reserved":0,"allocated":10555392,"data_stored":15225720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215122,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45424,"kb_used_data":14232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326416,"statfs":{"total":96636764160,"available":96590249984,"internally_reserved":0,"allocated":14573568,"data_stored":19731580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56420,"kb_used_data":25228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315420,"statfs":{"total":96636764160,"available":96578990080,"internally_reserved":0,"allocated":25833472,"data_stored":41549940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247826,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48032,"kb_used_data":12676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323808,"statfs":{"total":96636764160,"available":96587579392,"internally_reserved":0,"allocated":12980224,"data_stored":16612339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247826,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40104,"kb_used_data":8908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331736,"statfs":{"total":96636764160,"available":96595697664,"internally_reserved":0,"allocated":9121792,"data_stored":13003238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51340,"kb_used_data":15980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320500,"statfs":{"total":96636764160,"available":96584192000,"internally_reserved":0,"allocated":16363520,"data_stored":28289491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247824,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44512,"kb_used_data":13320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327328,"statfs":{"total":96636764160,"available":96591183872,"internally_reserved":0,"allocated":13639680,"data_stored":20443252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26349568,"data_stored":43720704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15114240,"data_stored":22708224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12869632,"data_stored":18534400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7962624,"data_stored":12345344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20779008,"data_stored":30543872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15208448,"data_stored":27631616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21598208,"data_stored":36110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13053952,"data_stored":20348928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":22536192,"data_stored":36052992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8470528,"data_stored":12791808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":17711104,"data_stored":29216768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16662528,"data_stored":27811840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":15167488,"data_stored":26865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":11292672,"data_stored":15425536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12619776,"data_stored":18157568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10502144,"data_stored":15859712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:02:12.094 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-16T04:02:12.116 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:12.116 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:02:12.116 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:02:12.116 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:12.130 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:02:12.132 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:12.128+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:12.232 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:12.228+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:12.310 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:12.311 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:02:12.311 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:02:12.311 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:02:12.311 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:12.322 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:12.322 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:12.327 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:02:12.333 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:12.329+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:12.433 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:12.429+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:12.449 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:12.449 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:02:12.449 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:02:12.449 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:02:12.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.449 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 54/33 ECSubWrite(tid=102, reqid=osd.4.0:102, at_version=54'46, trim_to=0'0, roll_forward_to=52'42))", 2024-09-16T04:02:12.449 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.801610+0000", 2024-09-16T04:02:12.449 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.643495080000001, 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011045743, 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801610+0000", 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.450 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801610+0000", 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801611+0000", 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6600000000000002e-07 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801635+0000", 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3977000000000001e-05 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.451 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801646+0000", 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1826e-05 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.801655+0000", 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7860000000000002e-06 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.806826+0000", 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051708489999999999 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.806837+0000", 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1153e-05 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812609+0000", 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0057714669999999997 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.812656+0000", 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7219000000000003e-05 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.453 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 75/33 ECSubWrite(tid=282, reqid=client.4646.0:706, at_version=75'166, trim_to=0'0, roll_forward_to=73'164))", 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.035830+0000", 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.409275069, 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.013088747, 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.035830+0000", 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.035830+0000", 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.035831+0000", 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2200000000000004e-07 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041273+0000", 2024-09-16T04:02:12.455 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0054425489999999997 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041291+0000", 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7337e-05 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041337+0000", 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6239000000000002e-05 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041462+0000", 2024-09-16T04:02:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012525700000000001 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041473+0000", 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1267000000000001e-05 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.048872+0000", 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0073982270000000003 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.048919+0000", 2024-09-16T04:02:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7348999999999997e-05 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 77/33 ECSubWrite(tid=195, reqid=osd.4.0:195, at_version=77'102, trim_to=0'0, roll_forward_to=76'98))", 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.734264+0000", 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7108415780000001, 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011117719, 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.458 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.734264+0000", 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.734264+0000", 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.734266+0000", 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0430000000000002e-06 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.734398+0000", 2024-09-16T04:02:12.459 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013187100000000001 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.734431+0000", 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3243e-05 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.734453+0000", 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2563000000000001e-05 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.460 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.734658+0000", 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020420599999999999 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.734685+0000", 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6906999999999998e-05 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.745288+0000", 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010603784999999999 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.461 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.745381+0000", 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3101000000000002e-05 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 77/33 ECSubWrite(tid=307, reqid=osd.11.0:307, at_version=77'189, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.737006+0000", 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7080991569999995, 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.013605815, 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737006+0000", 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737006+0000", 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737007+0000", 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7700000000000004e-07 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737062+0000", 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5509e-05 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737095+0000", 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2236000000000001e-05 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.464 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737118+0000", 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2963e-05 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737242+0000", 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000124914 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737267+0000", 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4473000000000001e-05 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.465 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.750525+0000", 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.013258452 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.750612+0000", 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6590999999999998e-05 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.466 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs6 77/33 ECSubWrite(tid=308, reqid=osd.11.0:308, at_version=77'60, trim_to=0'0, roll_forward_to=73'58))", 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.737659+0000", 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7074462520000004, 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.019522121, 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737659+0000", 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.467 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737659+0000", 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737661+0000", 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8899999999999999e-06 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737740+0000", 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9171999999999993e-05 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737772+0000", 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1479999999999997e-05 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.737793+0000", 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1509e-05 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738481+0000", 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000688198 2024-09-16T04:02:12.469 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738504+0000", 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2940000000000001e-05 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.757095+0000", 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.018590466999999999 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.757181+0000", 2024-09-16T04:02:12.470 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6465000000000006e-05 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 77/33 ECSubWrite(tid=309, reqid=osd.11.0:309, at_version=77'191, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.738067+0000", 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7070385720000001, 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015725711, 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.471 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738067+0000", 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738067+0000", 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738068+0000", 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.663e-06 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738114+0000", 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5408999999999998e-05 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738140+0000", 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6129e-05 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738161+0000", 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1027e-05 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738364+0000", 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020296299999999999 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738388+0000", 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4145999999999999e-05 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.753700+0000", 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015312212 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.753792+0000", 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2162000000000003e-05 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s5 77/33 ECSubWrite(tid=53, reqid=osd.15.0:53, at_version=77'43, trim_to=0'0, roll_forward_to=73'41))", 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.742555+0000", 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7025504520000005, 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.022699354000000001, 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.475 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742555+0000", 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742555+0000", 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742555+0000", 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0999999999999998e-07 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742616+0000", 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0256e-05 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742648+0000", 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2279e-05 2024-09-16T04:02:12.477 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742671+0000", 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2932999999999999e-05 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743718+0000", 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010467320000000001 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743743+0000", 2024-09-16T04:02:12.478 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5369000000000001e-05 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765164+0000", 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.021420903000000002 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765254+0000", 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0272000000000006e-05 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.479 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs6 77/33 ECSubWrite(tid=310, reqid=osd.11.0:310, at_version=77'62, trim_to=0'0, roll_forward_to=73'58))", 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.742676+0000", 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7024288040000002, 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.020760491999999998, 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.480 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742676+0000", 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742676+0000", 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742679+0000", 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.74e-06 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742782+0000", 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000103195 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742831+0000", 2024-09-16T04:02:12.482 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8758999999999999e-05 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742855+0000", 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3600000000000001e-05 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.748946+0000", 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0060913620000000003 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.483 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.748974+0000", 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8155000000000001e-05 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763356+0000", 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014381679 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763437+0000", 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1001999999999995e-05 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.484 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds4 77/33 ECSubWrite(tid=80, reqid=osd.7.0:80, at_version=77'57, trim_to=0'0, roll_forward_to=76'55))", 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.742783+0000", 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7023225360000005, 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.022541619999999998, 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742783+0000", 2024-09-16T04:02:12.485 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742783+0000", 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742783+0000", 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7700000000000004e-07 2024-09-16T04:02:12.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742834+0000", 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0254999999999999e-05 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742860+0000", 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6414000000000002e-05 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742879+0000", 2024-09-16T04:02:12.487 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9202e-05 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.744013+0000", 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001133457 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.744035+0000", 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.243e-05 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.488 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765263+0000", 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.021227772999999998 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765324+0000", 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1512000000000003e-05 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds4 77/33 ECSubWrite(tid=81, reqid=osd.7.0:81, at_version=77'59, trim_to=0'0, roll_forward_to=76'55))", 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.742911+0000", 2024-09-16T04:02:12.489 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7021940509999993, 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.027749737999999999, 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742911+0000", 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742911+0000", 2024-09-16T04:02:12.490 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742912+0000", 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.08e-06 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742980+0000", 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7961999999999997e-05 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743081+0000", 2024-09-16T04:02:12.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000101041 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743138+0000", 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6876000000000002e-05 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752959+0000", 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0098206950000000008 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.492 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752979+0000", 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0259999999999999e-05 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.770556+0000", 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.017576768999999999 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.770661+0000", 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000105055 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.493 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s5 77/33 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=77'45, trim_to=0'0, roll_forward_to=73'41))", 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.743321+0000", 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.7017844120000003, 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.023958719, 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743321+0000", 2024-09-16T04:02:12.494 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743321+0000", 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743322+0000", 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.497e-06 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.495 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743423+0000", 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000100498 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743460+0000", 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7582e-05 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743480+0000", 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9884000000000001e-05 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.496 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752812+0000", 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0093317769999999994 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752837+0000", 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4793000000000001e-05 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.767184+0000", 2024-09-16T04:02:12.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014347276000000001 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.767280+0000", 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5412000000000001e-05 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 77/33 ECSubWrite(tid=22, reqid=osd.5.0:22, at_version=77'17, trim_to=0'0, roll_forward_to=69'15))", 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.750699+0000", 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.6944064329999993, 2024-09-16T04:02:12.498 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012632269999999999, 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.750699+0000", 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.750699+0000", 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.750700+0000", 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8999999999999996e-07 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.750754+0000", 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4345999999999998e-05 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.750781+0000", 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7302000000000001e-05 2024-09-16T04:02:12.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.750802+0000", 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0784e-05 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.751031+0000", 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022921199999999999 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.751057+0000", 2024-09-16T04:02:12.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5746e-05 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763233+0000", 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012176004000000001 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763331+0000", 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8185999999999998e-05 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 77/33 ECSubWrite(tid=198, reqid=osd.4.0:198, at_version=77'104, trim_to=0'0, roll_forward_to=77'102))", 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.754665+0000", 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.6904400200000005, 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016009339000000001, 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754665+0000", 2024-09-16T04:02:12.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754665+0000", 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754667+0000", 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.75e-06 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754725+0000", 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7692000000000003e-05 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754758+0000", 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3133000000000003e-05 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754780+0000", 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.228e-05 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.755000+0000", 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021997600000000001 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.755025+0000", 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5153000000000001e-05 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.770587+0000", 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015562147 2024-09-16T04:02:12.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.770675+0000", 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7207999999999996e-05 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.4s4 77/33 ECSubWrite(tid=76, reqid=osd.8.0:76, at_version=77'46, trim_to=0'0, roll_forward_to=69'42))", 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.758997+0000", 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.6861081329999994, 2024-09-16T04:02:12.507 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011706991999999999, 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758997+0000", 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758997+0000", 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758999+0000", 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9400000000000001e-06 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.759051+0000", 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2389000000000003e-05 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.759096+0000", 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4079000000000003e-05 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.759116+0000", 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.02e-05 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.761596+0000", 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0024798820000000001 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.761621+0000", 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5262999999999999e-05 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.770620+0000", 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0089989639999999999 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.770704+0000", 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4275000000000002e-05 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.511 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs1 77/33 ECSubWrite(tid=199, reqid=osd.4.0:199, at_version=77'106, trim_to=0'0, roll_forward_to=77'102))", 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.761027+0000", 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.6840780189999993, 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012993930000000001, 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.761027+0000", 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.761027+0000", 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.761029+0000", 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.826e-06 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.761106+0000", 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6995000000000004e-05 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.761135+0000", 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9240000000000001e-05 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.761157+0000", 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1725999999999998e-05 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765844+0000", 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0046869340000000002 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765868+0000", 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4323000000000001e-05 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.773929+0000", 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0080605639999999992 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.774021+0000", 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2322000000000001e-05 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 77/33 ECSubWrite(tid=313, reqid=osd.11.0:313, at_version=77'195, trim_to=0'0, roll_forward_to=77'191))", 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.762782+0000", 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.6823236470000005, 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011243044000000001, 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.517 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.762782+0000", 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.762782+0000", 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.762784+0000", 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8759999999999999e-06 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.762834+0000", 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0176000000000002e-05 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.762867+0000", 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3692999999999997e-05 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.762890+0000", 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2756e-05 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765676+0000", 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027863440000000001 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765702+0000", 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5873000000000001e-05 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.773948+0000", 2024-09-16T04:02:12.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0082457339999999994 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.774025+0000", 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6592000000000001e-05 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs6 77/33 ECSubWrite(tid=315, reqid=osd.11.0:315, at_version=77'64, trim_to=0'0, roll_forward_to=77'62))", 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.766315+0000", 2024-09-16T04:02:12.521 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.6787903699999998, 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012906666000000001, 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.766315+0000", 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.766315+0000", 2024-09-16T04:02:12.522 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.766317+0000", 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2570000000000002e-06 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.766389+0000", 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1982000000000003e-05 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.766425+0000", 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5732999999999998e-05 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.766448+0000", 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3096e-05 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.766675+0000", 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022669300000000001 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.524 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.766702+0000", 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7509e-05 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.779136+0000", 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012433380000000001 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.779222+0000", 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6015999999999998e-05 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.525 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs6 77/33 ECSubWrite(tid=316, reqid=osd.11.0:316, at_version=77'66, trim_to=0'0, roll_forward_to=77'62))", 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.768457+0000", 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.676648771, 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012352742, 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.768457+0000", 2024-09-16T04:02:12.526 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.768457+0000", 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.768458+0000", 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.77e-06 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.768553+0000", 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4949000000000002e-05 2024-09-16T04:02:12.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.768618+0000", 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4862000000000003e-05 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.768641+0000", 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3045999999999999e-05 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.770849+0000", 2024-09-16T04:02:12.528 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022075879999999999 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.770874+0000", 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4865e-05 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.780725+0000", 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0098512259999999994 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.529 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.780809+0000", 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4436000000000003e-05 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.4s4 81/33 ECSubWrite(tid=82, reqid=osd.8.0:82, at_version=81'52, trim_to=0'0, roll_forward_to=80'48))", 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:06.736578+0000", 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.7085272719999995, 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011610143999999999, 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:12.530 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.736578+0000", 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.736578+0000", 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.736579+0000", 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.085e-06 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.531 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.736626+0000", 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6944999999999999e-05 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.736646+0000", 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0083000000000001e-05 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.736660+0000", 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3624e-05 2024-09-16T04:02:12.532 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.738850+0000", 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0021903790000000001 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.738863+0000", 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2418e-05 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.533 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.748149+0000", 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0092865919999999998 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.748188+0000", 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9017999999999998e-05 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:888 3.6s0 3:68ccc4ef:::smithi18831615-26:head [stat,delete] snapc 23=[23,21,20,1f,1e,1d,1b,19,18,17,16,15,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:11.732026+0000", 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.71307941200000002, 2024-09-16T04:02:12.534 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016086873000000002, 2024-09-16T04:02:12.535 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:12.535 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:12.535 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:12.535 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:12.535 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:12.535 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:12.535 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 888 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.732026+0000", 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.732026+0000", 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.536 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.732029+0000", 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9629999999999998e-06 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.732042+0000", 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3593e-05 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.732092+0000", 2024-09-16T04:02:12.537 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9653e-05 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.732109+0000", 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7133e-05 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.732200+0000", 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1132999999999996e-05 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.732210+0000", 2024-09-16T04:02:12.538 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9529999999999994e-06 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.736943+0000", 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00473275 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.737130+0000", 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000187061 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.738087+0000", 2024-09-16T04:02:12.539 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00095641599999999997 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.741662+0000", 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035756170000000001 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.748019+0000", 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0063567900000000002 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.748113+0000", 2024-09-16T04:02:12.540 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3811e-05 2024-09-16T04:02:12.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.541 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:12.541 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:12.541 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:12.542 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:02:12.543 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:12.540+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:12.644 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:12.637+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:12.716 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:12.716 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:02:12.716 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:02:12.717 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:12.730 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:02:12.745 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:12.740+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:12.811 INFO:tasks.rados.rados.0.smithi188.stdout:342: done (3 left) 2024-09-16T04:02:12.811 INFO:tasks.rados.rados.0.smithi188.stdout:343: done (2 left) 2024-09-16T04:02:12.812 INFO:tasks.rados.rados.0.smithi188.stdout:344: done (1 left) 2024-09-16T04:02:12.812 INFO:tasks.rados.rados.0.smithi188.stdout:345: done (0 left) 2024-09-16T04:02:12.812 INFO:tasks.rados.rados.0.smithi188.stdout:346: snap_remove snap 5 2024-09-16T04:02:12.827 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:02:12.846 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:12.844+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:12.858 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:12.858 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:02:12.858 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:02:12.859 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:02:12.859 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:12.874 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:02:12.947 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:12.944+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:363 3.bs0 3:d42ef308:::smithi18831615-39:head [rollback 5] snapc b=[b,a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 37.296429150999998, 2024-09-16T04:02:13.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01339692, 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 363 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:02:13.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746362+0000", 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746363+0000", 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.55e-06 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.050 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746372+0000", 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9900000000000003e-06 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746396+0000", 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.357e-05 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746414+0000", 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8012999999999998e-05 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750473+0000", 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040591439999999998 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750763+0000", 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000290165 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.752228+0000", 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014641910000000001 2024-09-16T04:02:13.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758077+0000", 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0058492529999999996 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.758283+0000", 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020590699999999999 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.759759+0000", 2024-09-16T04:02:13.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001476137 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:378 3.bs0 3:de226c4e:::smithi18831615-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 36.223293163999998, 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.013087019, 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.054 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 378 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.055 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819498+0000", 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819499+0000", 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1710000000000001e-06 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819503+0000", 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9319999999999998e-06 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.056 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819519+0000", 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5939000000000001e-05 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.819528+0000", 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9390000000000003e-06 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823625+0000", 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040971779999999999 2024-09-16T04:02:13.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.823698+0000", 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3417999999999994e-05 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.829121+0000", 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0054222879999999999 2024-09-16T04:02:13.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.829385+0000", 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026470200000000001 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:36.832585+0000", 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031994520000000002 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:423 3.bs0 3:da0cb12f:::smithi18831615-18:head [copy-from ver 9] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 33.238215562999997, 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.032968130999999998, 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 423 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.060 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804576+0000", 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804577+0000", 2024-09-16T04:02:13.061 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9759999999999998e-06 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804585+0000", 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6890000000000004e-06 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804602+0000", 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6912999999999999e-05 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804611+0000", 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7600000000000008e-06 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804668+0000", 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7521e-05 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804745+0000", 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6137999999999995e-05 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.832170+0000", 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.027425393999999999 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834467+0000", 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022970199999999999 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.837427+0000", 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002960074 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.837544+0000", 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011664599999999999 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.065 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:424 3.bs0 3:da0cb12f:::smithi18831615-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 33.238161679000001, 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.029561375000000001, 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 424 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.066 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804629+0000", 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804630+0000", 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6400000000000002e-07 2024-09-16T04:02:13.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804633+0000", 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0800000000000002e-06 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804642+0000", 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2520000000000005e-06 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804650+0000", 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3590000000000002e-06 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804908+0000", 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025827199999999998 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:13.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.804983+0000", 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5432000000000006e-05 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.833934+0000", 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.028950975 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834053+0000", 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000118553 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:39.834191+0000", 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013788799999999999 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:537 3.bs0 3:d947dc37:::smithi18831615-12:head [copy-from ver 116] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:13.071 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.931307+0000", 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.111484469000001, 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.017397910999999999, 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 537 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.072 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931307+0000", 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931307+0000", 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931309+0000", 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6400000000000001e-06 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931323+0000", 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3409e-05 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931352+0000", 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9346999999999999e-05 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931370+0000", 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7819e-05 2024-09-16T04:02:13.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931552+0000", 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018190200000000001 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931712+0000", 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000159918 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.942260+0000", 2024-09-16T04:02:13.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010547948 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.947604+0000", 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053446819999999999 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.948629+0000", 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010242859999999999 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.076 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.948705+0000", 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5959999999999998e-05 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.3s2 66/33 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=66'32, trim_to=0'0, roll_forward_to=63'30))", 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.737678+0000", 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 21.305113339999998, 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012970337, 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.077 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737678+0000", 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737678+0000", 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737678+0000", 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8299999999999998e-07 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737719+0000", 2024-09-16T04:02:13.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0689999999999998e-05 2024-09-16T04:02:13.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.080 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737735+0000", 2024-09-16T04:02:13.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6696000000000001e-05 2024-09-16T04:02:13.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737746+0000", 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0886999999999999e-05 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737879+0000", 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000132767 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.737906+0000", 2024-09-16T04:02:13.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7336000000000001e-05 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.750611+0000", 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012704317999999999 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.750648+0000", 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7259999999999999e-05 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.3s2 66/33 ECSubWrite(tid=240, reqid=osd.13.0:240, at_version=66'34, trim_to=0'0, roll_forward_to=63'30))", 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.738724+0000", 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 21.304066613, 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014283240000000001, 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738724+0000", 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738724+0000", 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738725+0000", 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4099999999999998e-07 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738747+0000", 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1988999999999999e-05 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738762+0000", 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4851999999999999e-05 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738770+0000", 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8329999999999994e-06 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738850+0000", 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9889999999999996e-05 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.738866+0000", 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6643999999999999e-05 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.751814+0000", 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01294794 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.753008+0000", 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001193351 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:631 3.bs0 3:de226c4e:::smithi18831615-16:13 [read 0~2539520,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T04:02:13.087 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:56.936888+0000", 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.105903375, 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01405375, 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 631 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.088 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.936888+0000", 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.936888+0000", 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.936888+0000", 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9999999999999997e-07 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.936893+0000", 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9180000000000002e-06 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.936904+0000", 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0926e-05 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.936912+0000", 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.515e-06 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.937050+0000", 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000138038 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.937056+0000", 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0809999999999999e-06 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.940715+0000", 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003659397 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.940824+0000", 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000108825 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:56.950941+0000", 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01011745 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.092 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:739 3.bs0 3:da0cb12f:::smithi18831615-18:head [copy-from ver 16] snapc 1e=[1e,1d,1c,1b,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.812925+0000", 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.229866031, 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.031207014000000002, 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 739 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.812925+0000", 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.812925+0000", 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.812926+0000", 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2130000000000001e-06 2024-09-16T04:02:13.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.812937+0000", 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.027e-05 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.812952+0000", 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5417000000000002e-05 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813233+0000", 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028101500000000002 2024-09-16T04:02:13.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816360+0000", 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031267479999999999 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816459+0000", 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9724999999999998e-05 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.835570+0000", 2024-09-16T04:02:13.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.019111060999999999 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.842392+0000", 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0068215910000000001 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.844007+0000", 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001615076 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.844132+0000", 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000124898 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:740 3.bs0 3:da0cb12f:::smithi18831615-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.813276+0000", 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.229515430999999, 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.026466474, 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.098 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 740 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813276+0000", 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813276+0000", 2024-09-16T04:02:13.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813277+0000", 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7770000000000001e-06 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813289+0000", 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1467e-05 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813316+0000", 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6855e-05 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813333+0000", 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7334e-05 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816737+0000", 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003403554 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816789+0000", 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2269999999999999e-05 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.839315+0000", 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.022526220999999999 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.839545+0000", 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022967199999999999 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.839742+0000", 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000197324 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.3s2 77/33 ECSubWrite(tid=271, reqid=osd.13.0:270, at_version=0'0, trim_to=0'0, roll_forward_to=77'57))", 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.752175+0000", 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.290615983, 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.017128833, 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.103 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752175+0000", 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752175+0000", 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752176+0000", 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.376e-06 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752200+0000", 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3954000000000001e-05 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752228+0000", 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7699e-05 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.752248+0000", 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9833000000000001e-05 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763361+0000", 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.01111295 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763384+0000", 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3203000000000001e-05 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.769023+0000", 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.005638573 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.769304+0000", 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028124500000000001 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:823 3.bs0 3:d084e290:::smithi18831615-9:head [copy-from ver 26] snapc 21=[21,20,1f,1e,1d,1c,1b,19,18,17,16,15,14,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:06.831042+0000", 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.2117494720000002, 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016778960999999998, 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 823 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.108 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831042+0000", 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831042+0000", 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831044+0000", 2024-09-16T04:02:13.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1560000000000002e-06 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831055+0000", 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1623e-05 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831089+0000", 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3800999999999997e-05 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831099+0000", 2024-09-16T04:02:13.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0199e-05 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831201+0000", 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000101421 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831328+0000", 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012745899999999999 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.842697+0000", 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011368724 2024-09-16T04:02:13.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.845758+0000", 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003060817 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.847758+0000", 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020004229999999999 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.847821+0000", 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2337999999999998e-05 2024-09-16T04:02:13.112 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:824 3.bs0 3:d084e290:::smithi18831615-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e81)", 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:06.831125+0000", 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.2116661930000001, 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014185247999999999, 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.113 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 824 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831125+0000", 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831125+0000", 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831125+0000", 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0500000000000003e-07 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831130+0000", 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4039999999999997e-06 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831139+0000", 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3209999999999994e-06 2024-09-16T04:02:13.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831146+0000", 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.319e-06 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831550+0000", 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00040467500000000001 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:13.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.831600+0000", 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9815000000000003e-05 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.845114+0000", 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.013513654999999999 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.845179+0000", 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5318e-05 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.845310+0000", 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013113600000000001 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.3s2 82/33 ECSubWrite(tid=285, reqid=osd.13.0:285, at_version=82'63, trim_to=0'0, roll_forward_to=81'61))", 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:07.774187+0000", 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.2686042579999999, 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014701594, 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.118 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.774187+0000", 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.774187+0000", 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.774187+0000", 2024-09-16T04:02:13.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0100000000000001e-07 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.774212+0000", 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4896999999999999e-05 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.774224+0000", 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2062000000000001e-05 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.774233+0000", 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2799999999999992e-06 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.774434+0000", 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020110899999999999 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.774445+0000", 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0172e-05 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.788792+0000", 2024-09-16T04:02:13.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014347607 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.788888+0000", 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6166000000000001e-05 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:826 3.bs0 3:da0cb12f:::smithi18831615-18:head [read 0~2342912,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:07.775252+0000", 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.2675394320000004, 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.034943325999999997, 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 826 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775252+0000", 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775252+0000", 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775254+0000", 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.951e-06 2024-09-16T04:02:13.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775261+0000", 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.9269999999999999e-06 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775281+0000", 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0811000000000001e-05 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775290+0000", 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0750000000000004e-06 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.780117+0000", 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048270159999999999 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.780443+0000", 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00032545399999999998 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.790663+0000", 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010220320999999999 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.790761+0000", 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8181000000000001e-05 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.810195+0000", 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.019433590000000001 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.129 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:827 3.bs0 3:da0cb12f:::smithi18831615-18:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:07.775308+0000", 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.267482684, 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.036063945, 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.130 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 827 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775308+0000", 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775308+0000", 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775309+0000", 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775314+0000", 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0340000000000004e-06 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775322+0000", 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9959999999999993e-06 2024-09-16T04:02:13.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775328+0000", 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7169999999999996e-06 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.780558+0000", 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0052305440000000002 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.780790+0000", 2024-09-16T04:02:13.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023143600000000001 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.791117+0000", 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010326886 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.791197+0000", 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0631999999999996e-05 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.811372+0000", 2024-09-16T04:02:13.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.020175003 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:828 3.bs0 3:da0cb12f:::smithi18831615-18:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:07.775336+0000", 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.2674546600000003, 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.038308285999999997, 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.135 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 828 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775336+0000", 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775336+0000", 2024-09-16T04:02:13.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775337+0000", 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4000000000000002e-07 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775353+0000", 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6433e-05 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775362+0000", 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3999999999999992e-06 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.775368+0000", 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0809999999999999e-06 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.780879+0000", 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0055109410000000001 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:13.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.781026+0000", 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014690800000000001 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.791506+0000", 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010480003999999999 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.791585+0000", 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9086999999999997e-05 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813645+0000", 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.022059892000000001 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:898 3.1s0 3:845c1c62:::smithi18831615-32:head [read 0~4456448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:11.866876+0000", 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.175915112, 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015671876000000001, 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.140 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 898 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866876+0000", 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866876+0000", 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866879+0000", 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1030000000000002e-06 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866896+0000", 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6736999999999999e-05 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866931+0000", 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5321999999999998e-05 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866949+0000", 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8054e-05 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867084+0000", 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013460800000000001 2024-09-16T04:02:13.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867183+0000", 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9536000000000004e-05 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.882548+0000", 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015364516 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:899 3.1s0 3:845c1c62:::smithi18831615-32:head [read 0~4456448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:11.866995+0000", 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.175796354, 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.019501609999999999, 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 899 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866995+0000", 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866995+0000", 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.866996+0000", 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4300000000000002e-07 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867002+0000", 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1199999999999999e-06 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867019+0000", 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6914000000000001e-05 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867027+0000", 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4999999999999999e-06 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867498+0000", 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000470648 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867813+0000", 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00031500899999999999 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.886496+0000", 2024-09-16T04:02:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.018683576 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:900 3.1s0 3:845c1c62:::smithi18831615-32:head [read 0~4456448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:11.867072+0000", 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.1757191119999999, 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.022609718000000001, 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 900 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867072+0000", 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867072+0000", 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867073+0000", 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867078+0000", 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.49e-06 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867090+0000", 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.187e-05 2024-09-16T04:02:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.867098+0000", 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2300000000000008e-06 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.871398+0000", 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0043001690000000004 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.871548+0000", 2024-09-16T04:02:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000149118 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:11.889682+0000", 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.018134063999999998 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.153 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:13.154 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:13.041+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:13.155 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:02:13.156 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:13.152+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:13.256 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:13.252+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:13.285 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:13.285 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:02:13.285 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:02:13.285 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:13.299 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:02:13.357 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:13.353+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:13.426 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:13.426 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:02:13.426 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:02:13.426 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:02:13.426 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:13.440 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:02:13.458 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:13.456+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:13.560 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:13.556+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:13.568 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:13.568 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:331 3.4s0 3:2a076d46:::smithi18831615-43:head [copy-from ver 17] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 46.641702000999999, 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.031262501999999998, 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 331 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922451+0000", 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.570 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922453+0000", 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.341e-06 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922458+0000", 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4070000000000001e-06 2024-09-16T04:02:13.571 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922470+0000", 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2363e-05 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922476+0000", 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5380000000000002e-06 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922556+0000", 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0481000000000004e-05 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922641+0000", 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4303999999999998e-05 2024-09-16T04:02:13.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.946336+0000", 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.023695544999999998 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953385+0000", 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0070484689999999999 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953631+0000", 2024-09-16T04:02:13.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00024578499999999999 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.953714+0000", 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3269000000000005e-05 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:332 3.4s0 3:2a076d46:::smithi18831615-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 46.64147577, 2024-09-16T04:02:13.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.026354305000000001, 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 332 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:02:13.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922678+0000", 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5900000000000005e-07 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922697+0000", 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8867000000000001e-05 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922710+0000", 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2591e-05 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922720+0000", 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0353000000000001e-05 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922817+0000", 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6675000000000005e-05 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.922866+0000", 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9060000000000001e-05 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.948791+0000", 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.025925143000000001 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.948905+0000", 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011362300000000001 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:26.949032+0000", 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012703399999999999 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:359 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 37.817952837999997, 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015151255000000001, 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.581 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 359 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746200+0000", 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746203+0000", 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6529999999999998e-06 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746217+0000", 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3946e-05 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746250+0000", 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3269999999999998e-05 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746268+0000", 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.804e-05 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746410+0000", 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014167300000000001 2024-09-16T04:02:13.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746432+0000", 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1999999999999999e-05 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750758+0000", 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004325527 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.750901+0000", 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000143361 2024-09-16T04:02:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.761352+0000", 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010450785000000001 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:360 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 37.817841973999997, 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016051205999999998, 2024-09-16T04:02:13.586 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 360 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746311+0000", 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746313+0000", 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3200000000000001e-06 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746326+0000", 2024-09-16T04:02:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2874000000000001e-05 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746352+0000", 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6395000000000001e-05 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746372+0000", 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9584000000000001e-05 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746569+0000", 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019797599999999999 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746579+0000", 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753659+0000", 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0070800610000000003 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.753897+0000", 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023741600000000001 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.762363+0000", 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0084659499999999999 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:361 3.4s0 3:20e031d1:::smithi18831615-29:a [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T04:02:13.591 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 37.817750969000002, 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016661775, 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 361 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746402+0000", 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746403+0000", 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:02:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746412+0000", 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3229999999999998e-06 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746433+0000", 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1484e-05 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746450+0000", 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6252e-05 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746636+0000", 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018645399999999999 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.746645+0000", 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9560000000000003e-06 2024-09-16T04:02:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754590+0000", 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0079450430000000006 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.754769+0000", 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017916000000000001 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:35.763064+0000", 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0082949830000000006 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:536 3.4s0 3:2238ca44:::smithi18831615-10:head [append 0~0] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:50.925928+0000", 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.638225266999999, 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011857786, 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 536 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925928+0000", 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925928+0000", 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925930+0000", 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.55e-06 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925937+0000", 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4359999999999996e-06 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925957+0000", 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0435999999999999e-05 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.925976+0000", 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.806e-05 2024-09-16T04:02:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.930543+0000", 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045675589999999997 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.930665+0000", 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000121765 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.931251+0000", 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00058618500000000003 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.933728+0000", 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0024773989999999999 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.937681+0000", 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0039529489999999999 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:50.937786+0000", 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000104447 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 75/33 ECSubWrite(tid=282, reqid=client.4646.0:706, at_version=75'166, trim_to=0'0, roll_forward_to=73'164))", 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.036274+0000", 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.52787925, 2024-09-16T04:02:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010371523000000001, 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.036274+0000", 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.036274+0000", 2024-09-16T04:02:13.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.036275+0000", 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.9400000000000005e-07 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041285+0000", 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050100079999999998 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041303+0000", 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8610999999999999e-05 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041314+0000", 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0406e-05 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041378+0000", 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4543999999999997e-05 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.041388+0000", 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9909999999999997e-06 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.046597+0000", 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0052090920000000002 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.046646+0000", 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8177000000000003e-05 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.607 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 77/33 ECSubWrite(tid=307, reqid=osd.11.0:307, at_version=77'189, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.738495+0000", 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.825658774000001, 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015787322999999999, 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.608 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738495+0000", 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738495+0000", 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738495+0000", 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.06e-07 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738555+0000", 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9642999999999998e-05 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738609+0000", 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4318999999999999e-05 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738637+0000", 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7305999999999999e-05 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738785+0000", 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014822699999999999 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738810+0000", 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4929000000000001e-05 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754195+0000", 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015385812 2024-09-16T04:02:13.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754282+0000", 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6380999999999998e-05 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs4 77/33 ECSubWrite(tid=308, reqid=osd.11.0:308, at_version=77'60, trim_to=0'0, roll_forward_to=73'58))", 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.738686+0000", 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.825467111, 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015355440999999999, 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.612 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.613 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.613 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.613 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738686+0000", 2024-09-16T04:02:13.613 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738686+0000", 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738687+0000", 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4600000000000003e-07 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738748+0000", 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0625999999999999e-05 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738772+0000", 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4053000000000002e-05 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738790+0000", 2024-09-16T04:02:13.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8340000000000001e-05 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.739032+0000", 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000242115 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.739058+0000", 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6037000000000002e-05 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.616 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.753925+0000", 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014866669000000001 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754042+0000", 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000116655 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 77/33 ECSubWrite(tid=309, reqid=osd.11.0:309, at_version=77'191, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.738840+0000", 2024-09-16T04:02:13.617 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.825313253999999, 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.019555064, 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738840+0000", 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738840+0000", 2024-09-16T04:02:13.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738841+0000", 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.004e-06 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738888+0000", 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6655e-05 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738912+0000", 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4300000000000001e-05 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.738930+0000", 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8156e-05 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.747852+0000", 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0089223189999999997 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.747878+0000", 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5040000000000001e-05 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758327+0000", 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010449448 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758395+0000", 2024-09-16T04:02:13.621 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8141999999999999e-05 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds5 77/33 ECSubWrite(tid=80, reqid=osd.7.0:80, at_version=77'57, trim_to=0'0, roll_forward_to=76'55))", 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.739828+0000", 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.824325472, 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011575709, 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.622 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.739828+0000", 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.739828+0000", 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.739828+0000", 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4700000000000002e-07 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.739880+0000", 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1922000000000002e-05 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.739944+0000", 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3966000000000004e-05 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.739967+0000", 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2572999999999999e-05 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740142+0000", 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017531 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740168+0000", 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6386000000000001e-05 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.751307+0000", 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011138777000000001 2024-09-16T04:02:13.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.751404+0000", 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6328000000000003e-05 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s1 77/33 ECSubWrite(tid=53, reqid=osd.15.0:53, at_version=77'43, trim_to=0'0, roll_forward_to=73'41))", 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.740086+0000", 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.824067448999999, 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014560639, 2024-09-16T04:02:13.626 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740086+0000", 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740086+0000", 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740086+0000", 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9299999999999998e-07 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740143+0000", 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6478999999999999e-05 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740173+0000", 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9862999999999999e-05 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740195+0000", 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2529000000000001e-05 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740314+0000", 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000119128 2024-09-16T04:02:13.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.740337+0000", 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2943e-05 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754560+0000", 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014223015 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754647+0000", 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6188999999999998e-05 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.fs4 77/33 ECSubWrite(tid=310, reqid=osd.11.0:310, at_version=77'62, trim_to=0'0, roll_forward_to=73'58))", 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.742305+0000", 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.821848471999999, 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016023313000000001, 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.631 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742305+0000", 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742305+0000", 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742307+0000", 2024-09-16T04:02:13.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742390+0000", 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3221999999999997e-05 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742423+0000", 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2512999999999999e-05 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742445+0000", 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2869000000000001e-05 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743290+0000", 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00084465899999999999 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743314+0000", 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4303000000000001e-05 2024-09-16T04:02:13.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758215+0000", 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014900208999999999 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758328+0000", 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000113558 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds5 77/33 ECSubWrite(tid=81, reqid=osd.7.0:81, at_version=77'59, trim_to=0'0, roll_forward_to=76'55))", 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.742575+0000", 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.821578473000001, 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012162760999999999, 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742575+0000", 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742575+0000", 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742576+0000", 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.46e-06 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742664+0000", 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7761000000000002e-05 2024-09-16T04:02:13.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742695+0000", 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0849999999999998e-05 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742716+0000", 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0706000000000001e-05 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.638 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742871+0000", 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015573400000000001 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742902+0000", 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.061e-05 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754664+0000", 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011762204 2024-09-16T04:02:13.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.754738+0000", 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3436000000000006e-05 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.9s5 77/33 ECSubWrite(tid=143, reqid=osd.14.0:143, at_version=77'23, trim_to=0'0, roll_forward_to=69'21))", 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.742984+0000", 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.821169354, 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010422328999999999, 2024-09-16T04:02:13.640 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742984+0000", 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742984+0000", 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.742985+0000", 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6700000000000003e-07 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743066+0000", 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1261999999999995e-05 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743091+0000", 2024-09-16T04:02:13.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5006000000000001e-05 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743112+0000", 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0679e-05 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743547+0000", 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000435716 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743562+0000", 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5003e-05 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.753322+0000", 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0097601070000000005 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.753406+0000", 2024-09-16T04:02:13.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3888999999999996e-05 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.5s1 77/33 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=77'45, trim_to=0'0, roll_forward_to=73'41))", 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.743238+0000", 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.820915788000001, 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015238326, 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743238+0000", 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743238+0000", 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743239+0000", 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.203e-06 2024-09-16T04:02:13.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743309+0000", 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0486000000000002e-05 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743339+0000", 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0071999999999999e-05 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.743361+0000", 2024-09-16T04:02:13.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.141e-05 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.751367+0000", 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0080063779999999998 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.751389+0000", 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1732000000000001e-05 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758276+0000", 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0068867490000000002 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.758476+0000", 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020029600000000001 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 77/33 ECSubWrite(tid=313, reqid=osd.11.0:313, at_version=77'195, trim_to=0'0, roll_forward_to=77'191))", 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.763724+0000", 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.800429445000001, 2024-09-16T04:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0097847430000000003, 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763724+0000", 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763724+0000", 2024-09-16T04:02:13.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763725+0000", 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2240000000000001e-06 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763771+0000", 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6195000000000001e-05 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763803+0000", 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1729000000000001e-05 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763825+0000", 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1824e-05 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765281+0000", 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001456424 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:13.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765307+0000", 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5936e-05 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.773442+0000", 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0081351989999999992 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.773509+0000", 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6211999999999996e-05 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.653 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:836 3.4s0 3:2e4a2a62:::smithi18831615-31:head [read 0~1638400,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:08.772655+0000", 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.791498088, 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010651011, 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 836 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.654 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772655+0000", 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772655+0000", 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772658+0000", 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8770000000000002e-06 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772671+0000", 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3338e-05 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772698+0000", 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6923e-05 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772713+0000", 2024-09-16T04:02:13.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4868000000000001e-05 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772818+0000", 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000104345 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772829+0000", 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1542000000000001e-05 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.657 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.776720+0000", 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0038913200000000002 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.776912+0000", 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019189399999999999 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.783306+0000", 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0063939039999999997 2024-09-16T04:02:13.658 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:837 3.4s0 3:2e4a2a62:::smithi18831615-31:head [read 0~1638400,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:08.772762+0000", 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.7913908149999997, 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.012081509000000001, 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.659 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.660 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.660 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 837 2024-09-16T04:02:13.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.660 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772762+0000", 2024-09-16T04:02:13.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772762+0000", 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772764+0000", 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772771+0000", 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1289999999999998e-06 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772784+0000", 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2655e-05 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772794+0000", 2024-09-16T04:02:13.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0181e-05 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772952+0000", 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000158732 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772963+0000", 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0723999999999999e-05 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.777594+0000", 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0046306730000000001 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.777760+0000", 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000166014 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.784844+0000", 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0070841649999999999 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4646.0:838 3.4s0 3:2e4a2a62:::smithi18831615-31:head [read 0~1638400,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:08.772810+0000", 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.7913430100000003, 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.013484194999999999, 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4646", 2024-09-16T04:02:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 838 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772810+0000", 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772810+0000", 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772811+0000", 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0800000000000004e-07 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772816+0000", 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6959999999999996e-06 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772826+0000", 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0052e-05 2024-09-16T04:02:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.772836+0000", 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0001e-05 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.773022+0000", 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000186462 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.773032+0000", 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5559999999999996e-06 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.778395+0000", 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053629970000000004 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.778517+0000", 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012207299999999999 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:08.786294+0000", 2024-09-16T04:02:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0077777499999999999 2024-09-16T04:02:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:13.670 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:13.672 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:02:13.672 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:13.668+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:13.773 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:13.769+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:13.804 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:13.804 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:02:13.804 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-16T04:02:13.804 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:13.815 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-09-16T04:02:13.820 INFO:tasks.rados.rados.0.smithi188.stdout:346: done (0 left) 2024-09-16T04:02:13.820 INFO:tasks.rados.rados.0.smithi188.stdout:347: setattr oid 12 current snap is 35 2024-09-16T04:02:13.825 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:02:13.829 INFO:tasks.rados.rados.0.smithi188.stdout:348: delete oid 33 current snap is 35 2024-09-16T04:02:13.861 INFO:tasks.rados.rados.0.smithi188.stdout:348: done (1 left) 2024-09-16T04:02:13.861 INFO:tasks.rados.rados.0.smithi188.stdout:349: delete oid 28 current snap is 35 2024-09-16T04:02:13.864 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 12 v 149 (ObjNum 101 snap 21 seq_num 101) dirty exists 2024-09-16T04:02:13.873 INFO:tasks.rados.rados.0.smithi188.stdout:347: done (1 left) 2024-09-16T04:02:13.873 INFO:tasks.rados.rados.0.smithi188.stdout:349: done (0 left) 2024-09-16T04:02:13.873 INFO:tasks.rados.rados.0.smithi188.stdout:350: snap_create 2024-09-16T04:02:13.876 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:13.869+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:13.962 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:13.962 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-16T04:02:13.963 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-16T04:02:13.963 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-16T04:02:13.963 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:13.974 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:13.969+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:13.977 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:02:14.075 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:14.069+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:14.106 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:02:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-16T04:02:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-16T04:02:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-16T04:02:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 56/33 ECSubWrite(tid=106, reqid=osd.4.0:106, at_version=56'49, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:02:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.730457+0000", 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.371360062000001, 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0085327589999999991, 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730457+0000", 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730457+0000", 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730458+0000", 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4600000000000004e-07 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730516+0000", 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8106000000000002e-05 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730546+0000", 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0043e-05 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730569+0000", 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2878999999999999e-05 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730837+0000", 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026877999999999998 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.730865+0000", 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7236000000000001e-05 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.738863+0000", 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.007998794 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.738990+0000", 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012627499999999999 2024-09-16T04:02:14.110 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 56/33 ECSubWrite(tid=107, reqid=osd.4.0:107, at_version=56'51, trim_to=0'0, roll_forward_to=54'47))", 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:41.732929+0000", 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.368888374999997, 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0092828939999999999, 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.732929+0000", 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.732929+0000", 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.732930+0000", 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5969999999999999e-06 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.732985+0000", 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4951999999999998e-05 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733014+0000", 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8544e-05 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.733036+0000", 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1942000000000001e-05 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.738085+0000", 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050491980000000004 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.738121+0000", 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6542999999999999e-05 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.742115+0000", 2024-09-16T04:02:14.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0039939470000000003 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:41.742212+0000", 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6170999999999998e-05 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 66/33 ECSubWrite(tid=140, reqid=osd.4.0:140, at_version=66'67, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.728419+0000", 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.373398242, 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0082188199999999999, 2024-09-16T04:02:14.115 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728419+0000", 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728419+0000", 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728419+0000", 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.65e-07 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728451+0000", 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1736000000000002e-05 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728469+0000", 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7819e-05 2024-09-16T04:02:14.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728483+0000", 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4188999999999999e-05 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728942+0000", 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00045947400000000003 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.728960+0000", 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7805e-05 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.736576+0000", 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0076158750000000002 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.736638+0000", 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1556999999999994e-05 2024-09-16T04:02:14.119 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 66/33 ECSubWrite(tid=141, reqid=osd.4.0:141, at_version=66'69, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:01:51.730367+0000", 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.371449807000001, 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011069918999999999, 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730367+0000", 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730367+0000", 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730369+0000", 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:02:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730408+0000", 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8977000000000003e-05 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730425+0000", 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7597000000000001e-05 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730435+0000", 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5619999999999993e-06 2024-09-16T04:02:14.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730506+0000", 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0817999999999999e-05 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.730515+0000", 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6360000000000006e-06 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.741355+0000", 2024-09-16T04:02:14.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.010840116 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:01:51.741437+0000", 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1710000000000002e-05 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 75/33 ECSubWrite(tid=282, reqid=client.4646.0:706, at_version=75'166, trim_to=0'0, roll_forward_to=73'164))", 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.037368+0000", 2024-09-16T04:02:14.124 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.064449328, 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011712406999999999, 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.037368+0000", 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.037368+0000", 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.037368+0000", 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6e-07 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.044086+0000", 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0067181910000000001 2024-09-16T04:02:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.044109+0000", 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2552999999999999e-05 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.044120+0000", 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1704e-05 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.044203+0000", 2024-09-16T04:02:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2157000000000006e-05 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.044213+0000", 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0207000000000001e-05 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.049040+0000", 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004827207 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.049080+0000", 2024-09-16T04:02:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0127999999999999e-05 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 76/33 ECSubWrite(tid=284, reqid=client.4646.0:719, at_version=76'168, trim_to=0'0, roll_forward_to=75'166))", 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.748284+0000", 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.353533220999999, 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0056218550000000003, 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.748284+0000", 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.748284+0000", 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.748284+0000", 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7500000000000001e-07 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.748547+0000", 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026265300000000001 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.748569+0000", 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2549999999999999e-05 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.748586+0000", 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6262000000000001e-05 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.748743+0000", 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000156909 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.748758+0000", 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.56e-05 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.753863+0000", 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0051044259999999996 2024-09-16T04:02:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.753906+0000", 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3080000000000001e-05 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 76/33 ECSubWrite(tid=293, reqid=client.4646.0:734, at_version=76'176, trim_to=0'0, roll_forward_to=76'174))", 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.813259+0000", 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.288558088, 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0095029220000000005, 2024-09-16T04:02:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813259+0000", 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813259+0000", 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.813261+0000", 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.807e-06 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.814509+0000", 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001248168 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.814614+0000", 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010466100000000001 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.814714+0000", 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000100622 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.814896+0000", 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018142000000000001 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.814911+0000", 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5603e-05 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.822659+0000", 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0077477199999999996 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.822762+0000", 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000102921 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 76/33 ECSubWrite(tid=294, reqid=client.4646.0:735, at_version=76'177, trim_to=0'0, roll_forward_to=76'174))", 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.816024+0000", 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.28579255, 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069468300000000002, 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816024+0000", 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816024+0000", 2024-09-16T04:02:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816025+0000", 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0499999999999999e-06 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816060+0000", 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4079999999999999e-05 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816086+0000", 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6528000000000001e-05 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816099+0000", 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2872e-05 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816369+0000", 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027016300000000002 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.816381+0000", 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2024999999999999e-05 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.822854+0000", 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0064726710000000002 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.822971+0000", 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000117441 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.141 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 76/33 ECSubWrite(tid=295, reqid=client.4646.0:736, at_version=76'178, trim_to=0'0, roll_forward_to=76'174))", 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.817196+0000", 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.284620656, 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.005860072, 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.817196+0000", 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.817196+0000", 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.817197+0000", 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0640000000000001e-06 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.817222+0000", 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4774e-05 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.817239+0000", 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6685e-05 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.817258+0000", 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8834e-05 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.817425+0000", 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016698099999999999 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.817450+0000", 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5435999999999998e-05 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.822980+0000", 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0055294059999999997 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.823056+0000", 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6891999999999995e-05 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 76/33 ECSubWrite(tid=300, reqid=client.4646.0:748, at_version=76'182, trim_to=0'0, roll_forward_to=76'178))", 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.835515+0000", 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.266302124999999, 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0063530119999999999, 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.146 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.835515+0000", 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.835515+0000", 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.835517+0000", 2024-09-16T04:02:14.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7269999999999999e-06 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.835916+0000", 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00039891900000000002 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.835947+0000", 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1925999999999999e-05 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.835973+0000", 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5193000000000001e-05 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.836209+0000", 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023640600000000001 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.836237+0000", 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7492999999999999e-05 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.841797+0000", 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0055600429999999998 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.841868+0000", 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1304999999999996e-05 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 76/33 ECSubWrite(tid=188, reqid=client.4646.0:739, at_version=76'98, trim_to=0'0, roll_forward_to=76'96))", 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:01.836210+0000", 2024-09-16T04:02:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.265607141, 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0074066369999999998, 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.836210+0000", 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.836210+0000", 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.836212+0000", 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6840000000000001e-06 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.837625+0000", 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014132210000000001 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.837737+0000", 2024-09-16T04:02:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000112668 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.837764+0000", 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6299999999999999e-05 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.838414+0000", 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00065030100000000005 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.838439+0000", 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.516e-05 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.843533+0000", 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050937420000000001 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:01.843616+0000", 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3560999999999995e-05 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 77/33 ECSubWrite(tid=313, reqid=osd.11.0:313, at_version=77'195, trim_to=0'0, roll_forward_to=77'191))", 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.763488+0000", 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.338329154, 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0055645390000000003, 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763488+0000", 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763488+0000", 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763489+0000", 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8700000000000005e-07 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763533+0000", 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4399e-05 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763557+0000", 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3969999999999999e-05 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.763576+0000", 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9476000000000001e-05 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765174+0000", 2024-09-16T04:02:14.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015975620000000001 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.765197+0000", 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3079e-05 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.768947+0000", 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0037502379999999999 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.769052+0000", 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000105028 2024-09-16T04:02:14.158 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 77/33 ECSubWrite(tid=317, reqid=osd.11.0:313, at_version=0'0, trim_to=0'0, roll_forward_to=77'195))", 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.775923+0000", 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.325893824, 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0060111829999999998, 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.775923+0000", 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.775923+0000", 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.775925+0000", 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.725e-06 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.775946+0000", 2024-09-16T04:02:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0820000000000001e-05 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.775975+0000", 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9541999999999999e-05 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.775989+0000", 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4123e-05 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776140+0000", 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015077600000000001 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776169+0000", 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9026e-05 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.781843+0000", 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0056740760000000001 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.781934+0000", 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1094999999999994e-05 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 77/33 ECSubWrite(tid=200, reqid=osd.4.0:199, at_version=0'0, trim_to=0'0, roll_forward_to=77'106))", 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:02.776283+0000", 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.325534245, 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0072860809999999998, 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776283+0000", 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776283+0000", 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776284+0000", 2024-09-16T04:02:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5600000000000001e-06 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776309+0000", 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4442000000000001e-05 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776344+0000", 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4940000000000001e-05 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776408+0000", 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4648999999999997e-05 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776594+0000", 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000185921 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.776617+0000", 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3010000000000002e-05 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.783496+0000", 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.006878365 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:02.783569+0000", 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3194000000000004e-05 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 81/33 ECSubWrite(tid=218, reqid=osd.4.0:218, at_version=81'116, trim_to=0'0, roll_forward_to=79'112))", 2024-09-16T04:02:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:06.752675+0000", 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.349141522, 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.007457511, 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.752675+0000", 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.752675+0000", 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.752676+0000", 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6000000000000004e-07 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.752726+0000", 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9780000000000001e-05 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.752740+0000", 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4352000000000001e-05 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.752751+0000", 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0761000000000001e-05 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.753250+0000", 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00049870199999999996 2024-09-16T04:02:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.753261+0000", 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1533e-05 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.760034+0000", 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0067727990000000004 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.760133+0000", 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9024000000000006e-05 2024-09-16T04:02:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 81/33 ECSubWrite(tid=223, reqid=osd.4.0:223, at_version=81'120, trim_to=0'0, roll_forward_to=81'116))", 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:06.763924+0000", 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.3378925050000001, 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0057735340000000003, 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.763924+0000", 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.763924+0000", 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.763925+0000", 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.7400000000000003e-07 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.763951+0000", 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5643000000000002e-05 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.763964+0000", 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3570999999999999e-05 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.763973+0000", 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1109999999999997e-06 2024-09-16T04:02:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.764449+0000", 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000475685 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.764459+0000", 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7340000000000003e-06 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.769659+0000", 2024-09-16T04:02:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0052006159999999999 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:06.769698+0000", 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8600000000000003e-05 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 82/33 ECSubWrite(tid=359, reqid=osd.11.0:359, at_version=82'235, trim_to=0'0, roll_forward_to=82'233))", 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:07.813554+0000", 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.2882630620000004, 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0062107489999999998, 2024-09-16T04:02:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813554+0000", 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813554+0000", 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813555+0000", 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3699999999999999e-07 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813579+0000", 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4076000000000001e-05 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813592+0000", 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3499000000000001e-05 2024-09-16T04:02:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813612+0000", 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9091999999999999e-05 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813698+0000", 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6185000000000002e-05 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.179 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.813707+0000", 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3160000000000003e-06 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.819702+0000", 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0059947840000000004 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.819765+0000", 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2860000000000005e-05 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.180 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 82/33 ECSubWrite(tid=366, reqid=osd.11.0:366, at_version=82'241, trim_to=0'0, roll_forward_to=82'237))", 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:07.831021+0000", 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.2707962630000003, 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0057074170000000002, 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.181 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.831021+0000", 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.831021+0000", 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.831021+0000", 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4199999999999995e-07 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.182 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.831049+0000", 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7419e-05 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.831075+0000", 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5743999999999999e-05 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.831093+0000", 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8711000000000001e-05 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.831200+0000", 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000106883 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.831214+0000", 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3281e-05 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.836689+0000", 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0054757419999999996 2024-09-16T04:02:14.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:07.836728+0000", 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8894999999999999e-05 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.bs3 88/33 ECSubWrite(tid=284, reqid=osd.4.0:282, at_version=0'0, trim_to=0'0, roll_forward_to=88'155))", 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:13.877990+0000", 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.22382680399999999, 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0062629000000000001, 2024-09-16T04:02:14.185 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.877990+0000", 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.877990+0000", 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.186 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.877991+0000", 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7800000000000001e-07 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.877999+0000", 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9540000000000004e-06 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.187 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.878013+0000", 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3747999999999999e-05 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.878022+0000", 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0969999999999995e-06 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.878095+0000", 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3009000000000005e-05 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.188 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.878107+0000", 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2754e-05 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.884211+0000", 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0061036800000000002 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.884253+0000", 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1980000000000001e-05 2024-09-16T04:02:14.189 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 88/33 ECSubWrite(tid=393, reqid=osd.11.0:393, at_version=88'265, trim_to=0'0, roll_forward_to=88'261))", 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-16T04:02:13.882022+0000", 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.21979523300000001, 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0056616640000000003, 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-16T04:02:14.190 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.882022+0000", 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.882022+0000", 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.882022+0000", 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1699999999999998e-07 2024-09-16T04:02:14.191 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.882058+0000", 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5787999999999997e-05 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.882071+0000", 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2486e-05 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.882077+0000", 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0909999999999998e-06 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.882129+0000", 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1912e-05 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.882136+0000", 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.9240000000000001e-06 2024-09-16T04:02:14.193 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.887659+0000", 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0055237669999999997 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-16T04:02:13.887683+0000", 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4079000000000001e-05 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T04:02:14.194 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:02:14.196 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:02:14.197 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:14.192+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:14.297 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:14.296+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:14.334 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:14.334 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:14.334 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:02:14.335 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:14.345 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:02:14.349 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:02:14.398 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:14.396+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:14.470 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:14.470 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:14.470 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:02:14.470 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:02:14.470 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:14.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:02:14.503 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:14.504 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:14.522 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:14.522 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:14.534 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:14.532+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:14.543 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:14.565 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:14.565 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:14.582 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:14.582 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:14.600 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:14.596+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:14.602 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:14.603 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:14.615 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:14.616 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:14.625 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:14.625 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:14.630 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:02:14.647 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:14.647 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:14.663 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:14.664 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:14.683 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:14.683 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:14.700 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:14.696+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:14.706 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:14.706 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:14.728 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:14.746 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:14.767 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:14.789 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:14.801 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:14.801 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:14.802 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:14.800+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:14.811 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:14.811 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:14.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:02:14.822 INFO:tasks.rados.rados.0.smithi188.stdout:350: done (0 left) 2024-09-16T04:02:14.822 INFO:tasks.rados.rados.0.smithi188.stdout:351: copy_from oid 46 from oid 50 current snap is 36 2024-09-16T04:02:14.822 INFO:tasks.rados.rados.0.smithi188.stdout:352: read oid 44 snap -1 2024-09-16T04:02:14.822 INFO:tasks.rados.rados.0.smithi188.stdout:352: expect (ObjNum 69 snap 2 seq_num 69) 2024-09-16T04:02:14.822 INFO:tasks.rados.rados.0.smithi188.stdout:353: delete oid 10 current snap is 36 2024-09-16T04:02:14.831 INFO:tasks.rados.rados.0.smithi188.stdout:353: done (2 left) 2024-09-16T04:02:14.831 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 15 current snap is 36 2024-09-16T04:02:14.831 INFO:tasks.rados.rados.0.smithi188.stdout:354: seq_num 144 ranges {0=16384} 2024-09-16T04:02:14.831 INFO:tasks.rados.rados.0.smithi188.stdout:354: writing smithi18831615-15 from 0 to 16384 tid 1 2024-09-16T04:02:14.831 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 4 current snap is 36 2024-09-16T04:02:14.831 INFO:tasks.rados.rados.0.smithi188.stdout:355: seq_num 145 ranges {3375104=32768} 2024-09-16T04:02:14.885 INFO:tasks.rados.rados.0.smithi188.stdout:355: writing smithi18831615-4 from 3375104 to 3407872 tid 1 2024-09-16T04:02:14.885 INFO:tasks.rados.rados.0.smithi188.stdout:356: read oid 7 snap -1 2024-09-16T04:02:14.885 INFO:tasks.rados.rados.0.smithi188.stdout:356: expect (ObjNum 132 snap 32 seq_num 132) 2024-09-16T04:02:14.885 INFO:tasks.rados.rados.0.smithi188.stdout:357: snap_create 2024-09-16T04:02:14.886 INFO:tasks.rados.rados.0.smithi188.stdout:351: finishing copy_from to smithi18831615-46 2024-09-16T04:02:14.886 INFO:tasks.rados.rados.0.smithi188.stdout:351: got expected ENOENT (src dne) 2024-09-16T04:02:14.886 INFO:tasks.rados.rados.0.smithi188.stdout:351: finishing copy_from racing read to smithi18831615-46 2024-09-16T04:02:14.886 INFO:tasks.rados.rados.0.smithi188.stdout:354: finishing write tid 1 to smithi18831615-15 2024-09-16T04:02:14.886 INFO:tasks.rados.rados.0.smithi188.stdout:354: finishing write tid 2 to smithi18831615-15 2024-09-16T04:02:14.886 INFO:tasks.rados.rados.0.smithi188.stdout:352: expect (ObjNum 69 snap 2 seq_num 69) 2024-09-16T04:02:14.902 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:14.900+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:14.946 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:14.946 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:14.960 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:02:15.000 INFO:tasks.rados.rados.0.smithi188.stdout:354: finishing write tid 3 to smithi18831615-15 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:354: oid 15 updating version 0 to 19 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:354: oid 15 version 19 is already newer than 18 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 15 v 19 (ObjNum 144 snap 36 seq_num 144) dirty exists 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:354: left oid 15 (ObjNum 144 snap 36 seq_num 144) 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:355: finishing write tid 1 to smithi18831615-4 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:355: finishing write tid 2 to smithi18831615-4 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:355: finishing write tid 3 to smithi18831615-4 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:355: oid 4 updating version 0 to 268 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:355: oid 4 version 268 is already newer than 267 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 4 v 268 (ObjNum 145 snap 36 seq_num 145) dirty exists 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:355: left oid 4 (ObjNum 145 snap 36 seq_num 145) 2024-09-16T04:02:15.001 INFO:tasks.rados.rados.0.smithi188.stdout:356: expect (ObjNum 132 snap 32 seq_num 132) 2024-09-16T04:02:15.003 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:15.000+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:15.104 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:15.100+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:15.127 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:15.127 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:15.142 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:02:15.205 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:15.201+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:15.261 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:15.261 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:15.275 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:02:15.306 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:15.300+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:15.402 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:15.402 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:15.407 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:15.404+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:15.416 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:02:15.508 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:15.504+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:15.558 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:02:15.609 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:15.604+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:15.678 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:15.678 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:15.692 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:02:15.710 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:15.708+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:15.811 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:15.808+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:15.834 INFO:tasks.rados.rados.0.smithi188.stdout:351: done (5 left) 2024-09-16T04:02:15.834 INFO:tasks.rados.rados.0.smithi188.stdout:352: done (4 left) 2024-09-16T04:02:15.834 INFO:tasks.rados.rados.0.smithi188.stdout:354: done (3 left) 2024-09-16T04:02:15.834 INFO:tasks.rados.rados.0.smithi188.stdout:355: done (2 left) 2024-09-16T04:02:15.834 INFO:tasks.rados.rados.0.smithi188.stdout:356: done (1 left) 2024-09-16T04:02:15.834 INFO:tasks.rados.rados.0.smithi188.stdout:357: done (0 left) 2024-09-16T04:02:15.835 INFO:tasks.rados.rados.0.smithi188.stdout:358: rollback oid 26 current snap is 37 2024-09-16T04:02:15.835 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 26 to 11 2024-09-16T04:02:15.835 INFO:tasks.rados.rados.0.smithi188.stdout:359: snap_create 2024-09-16T04:02:15.845 INFO:tasks.rados.rados.0.smithi188.stdout:358: finishing rollback tid 1 to smithi18831615-26 2024-09-16T04:02:15.845 INFO:tasks.rados.rados.0.smithi188.stdout:358: finishing rollback tid 2 to smithi18831615-26 2024-09-16T04:02:15.846 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 26 v 8 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T04:02:15.848 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:02:15.866 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:15.866 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:15.882 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:02:15.912 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:15.909+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:16.002 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:16.002 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:16.013 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:16.008+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:16.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:02:16.115 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:16.112+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:16.149 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:16.150 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:16.164 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:02:16.216 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:16.212+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:16.287 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:16.287 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:16.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:02:16.317 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:16.312+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:16.418 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:16.416+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:16.423 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:16.423 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:16.438 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:02:16.519 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:16.516+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:16.561 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:16.562 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:16.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:02:16.620 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:16.616+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:16.698 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:16.698 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:16.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:02:16.729 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-09-16T04:02:16.734 INFO:tasks.rados.rados.0.smithi188.stdout:358: done (1 left) 2024-09-16T04:02:16.734 INFO:tasks.rados.rados.0.smithi188.stdout:359: done (0 left) 2024-09-16T04:02:16.734 INFO:tasks.rados.rados.0.smithi188.stdout:360: rollback oid 35 current snap is 38 2024-09-16T04:02:16.734 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 35 to 36 2024-09-16T04:02:16.734 INFO:tasks.rados.rados.0.smithi188.stdout:361: read oid 6 snap -1 2024-09-16T04:02:16.735 INFO:tasks.rados.rados.0.smithi188.stdout:361: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T04:02:16.735 INFO:tasks.rados.rados.0.smithi188.stdout:362: rollback oid 26 current snap is 38 2024-09-16T04:02:16.735 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 26 to 35 2024-09-16T04:02:16.735 INFO:tasks.rados.rados.0.smithi188.stdout:363: delete oid 16 current snap is 38 2024-09-16T04:02:16.750 INFO:tasks.rados.rados.0.smithi188.stdout:362: finishing rollback tid 0 to smithi18831615-26 2024-09-16T04:02:16.750 INFO:tasks.rados.rados.0.smithi188.stdout:362: finishing rollback tid 1 to smithi18831615-26 2024-09-16T04:02:16.750 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 26 v 11 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T04:02:16.752 INFO:tasks.rados.rados.0.smithi188.stdout:360: finishing rollback tid 0 to smithi18831615-35 2024-09-16T04:02:16.756 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:02:16.753+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:02:16.758 INFO:tasks.rados.rados.0.smithi188.stdout:361: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T04:02:16.817 INFO:tasks.rados.rados.0.smithi188.stdout:360: finishing rollback tid 1 to smithi18831615-35 2024-09-16T04:02:16.818 INFO:tasks.rados.rados.0.smithi188.stdout:360: finishing rollback tid 2 to smithi18831615-35 2024-09-16T04:02:16.818 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 103 (ObjNum 90 snap 16 seq_num 90) dirty exists 2024-09-16T04:02:16.818 INFO:tasks.rados.rados.0.smithi188.stdout:360: done (3 left) 2024-09-16T04:02:16.818 INFO:tasks.rados.rados.0.smithi188.stdout:361: done (2 left) 2024-09-16T04:02:16.818 INFO:tasks.rados.rados.0.smithi188.stdout:362: done (1 left) 2024-09-16T04:02:16.819 INFO:tasks.rados.rados.0.smithi188.stdout:363: done (0 left) 2024-09-16T04:02:16.819 INFO:tasks.rados.rados.0.smithi188.stdout:364: read oid 13 snap -1 2024-09-16T04:02:16.819 INFO:tasks.rados.rados.0.smithi188.stdout:364: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:02:16.819 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 44 current snap is 38 2024-09-16T04:02:16.819 INFO:tasks.rados.rados.0.smithi188.stdout:365: seq_num 147 ranges {1900544=65536} 2024-09-16T04:02:16.821 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:16.816+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:16.836 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:02:16.837 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:16.837 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:16.849 INFO:tasks.rados.rados.0.smithi188.stdout:365: writing smithi18831615-44 from 1900544 to 1966080 tid 1 2024-09-16T04:02:16.849 INFO:tasks.rados.rados.0.smithi188.stdout:366: copy_from oid 36 from oid 40 current snap is 38 2024-09-16T04:02:16.849 INFO:tasks.rados.rados.0.smithi188.stdout:364: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:02:16.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:02:16.854 INFO:tasks.rados.rados.0.smithi188.stdout:366: finishing copy_from to smithi18831615-36 2024-09-16T04:02:16.854 INFO:tasks.rados.rados.0.smithi188.stdout:366: got expected ENOENT (src dne) 2024-09-16T04:02:16.854 INFO:tasks.rados.rados.0.smithi188.stdout:366: finishing copy_from racing read to smithi18831615-36 2024-09-16T04:02:16.855 INFO:tasks.rados.rados.0.smithi188.stdout:364: done (2 left) 2024-09-16T04:02:16.855 INFO:tasks.rados.rados.0.smithi188.stdout:366: done (1 left) 2024-09-16T04:02:16.855 INFO:tasks.rados.rados.0.smithi188.stdout:367: rollback oid 11 current snap is 38 2024-09-16T04:02:16.855 INFO:tasks.rados.rados.0.smithi188.stdout:rollback oid 11 to 35 2024-09-16T04:02:16.855 INFO:tasks.rados.rados.0.smithi188.stdout:368: snap_remove snap 28 2024-09-16T04:02:16.922 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:16.920+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:16.993 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:17.014 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:17.014 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:17.014 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:17.015 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:17.022 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:17.020+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:17.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:02:17.034 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:17.034 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:17.054 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:17.055 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:17.074 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:17.074 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:17.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:02:17.095 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:02:17.095 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T04:02:17.095 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.057398 2024-09-16T04:02:17.096 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 0.05739804652845537 2024-09-16T04:02:17.096 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:17.096 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:17.116 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:17.116 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:17.123 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:17.120+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:17.138 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:17.138 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:17.149 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:17.150 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:17.156 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:17.164 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:02:17.178 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:17.197 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:17.221 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:17.221 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:17.223 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:02:17.217+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:02:17.235 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:17.235 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:17.270 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:17.271 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 75/33 ECSubWrite(tid=282, reqid=client.4646.0:706, at_version=75'166, trim_to=0'0, roll_forward_to=73'164))", 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:01.037358+0000", 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 16.249631098999998, 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012998964, 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.291 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.037358+0000", 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.037358+0000", 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.292 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.037358+0000", 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.66e-07 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.042178+0000", 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0048200309999999998 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.042194+0000", 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6169000000000001e-05 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.293 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.294 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.294 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.042204+0000", 2024-09-16T04:02:17.294 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.9289999999999993e-06 2024-09-16T04:02:17.294 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.294 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.294 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.294 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.042261+0000", 2024-09-16T04:02:17.294 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7519000000000003e-05 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.042269+0000", 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3950000000000003e-06 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.295 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.050288+0000", 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0080192140000000002 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.050357+0000", 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8541000000000007e-05 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.296 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 77/33 ECSubWrite(tid=310, reqid=osd.11.0:310, at_version=77'62, trim_to=0'0, roll_forward_to=73'58))", 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:02.743223+0000", 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 14.543765908999999, 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011281075999999999, 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743223+0000", 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.297 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743223+0000", 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743224+0000", 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.657e-06 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.298 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743312+0000", 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7437999999999999e-05 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743339+0000", 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7333000000000001e-05 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743358+0000", 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9093000000000002e-05 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.299 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743593+0000", 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023446799999999999 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743619+0000", 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6092999999999999e-05 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.754429+0000", 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010809890000000001 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.754504+0000", 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.5104000000000005e-05 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.301 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 81/33 ECSubWrite(tid=333, reqid=osd.11.0:333, at_version=81'211, trim_to=0'0, roll_forward_to=79'209))", 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.738584+0000", 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.548404591000001, 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014561236999999999, 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738584+0000", 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.302 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738584+0000", 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738585+0000", 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.47e-07 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738642+0000", 2024-09-16T04:02:17.303 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6622000000000001e-05 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738674+0000", 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1979e-05 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738698+0000", 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.402e-05 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.304 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739193+0000", 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00049564100000000001 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739220+0000", 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6421999999999999e-05 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753065+0000", 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013844881 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.305 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753145+0000", 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0724999999999997e-05 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 81/33 ECSubWrite(tid=334, reqid=osd.11.0:334, at_version=81'68, trim_to=0'0, roll_forward_to=77'66))", 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.739920+0000", 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.547068759, 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012342739, 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.306 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739920+0000", 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739920+0000", 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739922+0000", 2024-09-16T04:02:17.307 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.77e-06 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739974+0000", 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.1901999999999999e-05 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740005+0000", 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1086e-05 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740037+0000", 2024-09-16T04:02:17.308 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2693e-05 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740218+0000", 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00018028 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740243+0000", 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5063e-05 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.309 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.752184+0000", 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0119411 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.752263+0000", 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8844999999999995e-05 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 81/33 ECSubWrite(tid=335, reqid=osd.11.0:335, at_version=81'213, trim_to=0'0, roll_forward_to=79'209))", 2024-09-16T04:02:17.310 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.740712+0000", 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 10.546276401, 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015661727, 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740712+0000", 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.311 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740712+0000", 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740714+0000", 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7269999999999999e-06 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740785+0000", 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.0605000000000006e-05 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.312 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740814+0000", 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9209000000000001e-05 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740839+0000", 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5276999999999999e-05 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742346+0000", 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001506982 2024-09-16T04:02:17.313 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742371+0000", 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4922e-05 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.756263+0000", 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013892124 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.756374+0000", 2024-09-16T04:02:17.314 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000110881 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 82/33 ECSubWrite(tid=355, reqid=osd.11.0:355, at_version=82'74, trim_to=0'0, roll_forward_to=81'72))", 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.796123+0000", 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.4908659750000002, 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012782646999999999, 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.315 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796123+0000", 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796123+0000", 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796123+0000", 2024-09-16T04:02:17.316 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.0299999999999999e-07 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796175+0000", 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2156e-05 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796218+0000", 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.2339000000000002e-05 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.317 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796243+0000", 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4909000000000002e-05 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796534+0000", 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00029144499999999999 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796554+0000", 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0299e-05 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.318 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.808810+0000", 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012255448 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.808905+0000", 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5447999999999998e-05 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 82/33 ECSubWrite(tid=354, reqid=osd.11.0:354, at_version=82'231, trim_to=0'0, roll_forward_to=81'229))", 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.796295+0000", 2024-09-16T04:02:17.319 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.4906941549999999, 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012790187999999999, 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796295+0000", 2024-09-16T04:02:17.320 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796295+0000", 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796295+0000", 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.0599999999999999e-07 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.321 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796343+0000", 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7809000000000003e-05 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796366+0000", 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2770000000000001e-05 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796383+0000", 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7183000000000001e-05 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.322 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796528+0000", 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000144544 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796552+0000", 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4482999999999999e-05 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.808945+0000", 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012392535 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.323 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.809085+0000", 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000139958 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 82/33 ECSubWrite(tid=356, reqid=osd.11.0:356, at_version=82'76, trim_to=0'0, roll_forward_to=81'72))", 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.796927+0000", 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.4900619709999994, 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015245435999999999, 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.324 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796927+0000", 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796927+0000", 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.325 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796928+0000", 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.79e-06 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.796974+0000", 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.5235999999999999e-05 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.797001+0000", 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6965999999999999e-05 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.326 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.797022+0000", 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1299999999999999e-05 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.797610+0000", 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00058812399999999998 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.797769+0000", 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000158894 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.812104+0000", 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014335076 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.812172+0000", 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8050000000000001e-05 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.ds3 82/33 ECSubWrite(tid=96, reqid=client.4646.0:829, at_version=82'77, trim_to=0'0, roll_forward_to=81'71))", 2024-09-16T04:02:17.328 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.799176+0000", 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.4878131450000005, 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013338129000000001, 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799176+0000", 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.329 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799176+0000", 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799178+0000", 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.800690+0000", 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0015121100000000001 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.330 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.800722+0000", 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2069000000000001e-05 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.800760+0000", 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8485999999999997e-05 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801486+0000", 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00072537299999999997 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.331 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.801511+0000", 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5099000000000001e-05 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.812439+0000", 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010927917000000001 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.812514+0000", 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.5094999999999998e-05 2024-09-16T04:02:17.332 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 82/33 ECSubWrite(tid=357, reqid=osd.11.0:357, at_version=82'233, trim_to=0'0, roll_forward_to=81'229))", 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.799365+0000", 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 9.4876236449999993, 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012698687, 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.333 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799365+0000", 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799365+0000", 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799384+0000", 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8482e-05 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.334 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799438+0000", 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.4506000000000003e-05 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799470+0000", 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1633e-05 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799494+0000", 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3909000000000001e-05 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.335 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799949+0000", 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00045524399999999999 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.799974+0000", 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5054000000000001e-05 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.811985+0000", 2024-09-16T04:02:17.336 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012011109000000001 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.812064+0000", 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.8750000000000003e-05 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 88/33 ECSubWrite(tid=273, reqid=osd.4.0:273, at_version=88'51, trim_to=0'0, roll_forward_to=86'49))", 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.830808+0000", 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.4561811329999998, 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018138294999999999, 2024-09-16T04:02:17.337 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830808+0000", 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830808+0000", 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.338 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830808+0000", 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830867+0000", 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8315e-05 2024-09-16T04:02:17.339 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830918+0000", 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0878999999999999e-05 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830945+0000", 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.764e-05 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831414+0000", 2024-09-16T04:02:17.340 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00046899799999999998 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831442+0000", 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7509999999999999e-05 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.848852+0000", 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.017410164999999998 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.341 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.848946+0000", 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.3927999999999999e-05 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.ds3 88/33 ECSubWrite(tid=116, reqid=osd.7.0:116, at_version=88'95, trim_to=0'0, roll_forward_to=86'93))", 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.830932+0000", 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.4560571150000001, 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.016407600000000001, 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.342 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830932+0000", 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830932+0000", 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.343 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.830932+0000", 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.9599999999999998e-07 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831001+0000", 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8256000000000005e-05 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831035+0000", 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.4805999999999998e-05 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.344 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831060+0000", 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.446e-05 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831435+0000", 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00037465600000000001 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831461+0000", 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6743000000000001e-05 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.345 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.847262+0000", 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015800410000000001 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.847339+0000", 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7472999999999995e-05 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.ds3 88/33 ECSubWrite(tid=117, reqid=osd.7.0:117, at_version=88'97, trim_to=0'0, roll_forward_to=86'93))", 2024-09-16T04:02:17.346 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.831119+0000", 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.4558694050000001, 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.025659999999999999, 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831119+0000", 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831119+0000", 2024-09-16T04:02:17.347 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831120+0000", 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.003e-06 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831189+0000", 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.8771999999999998e-05 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.348 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831216+0000", 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6766000000000001e-05 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.831235+0000", 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9182999999999999e-05 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.837442+0000", 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0062073969999999999 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.349 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.837469+0000", 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6497000000000001e-05 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.856682+0000", 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019213023999999999 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.856779+0000", 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.7357999999999999e-05 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.350 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s4 88/33 ECSubWrite(tid=275, reqid=osd.4.0:275, at_version=88'53, trim_to=0'0, roll_forward_to=86'49))", 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.834358+0000", 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.4526309620000002, 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.023144303000000001, 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.351 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834358+0000", 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834358+0000", 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834360+0000", 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.04e-06 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.352 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834414+0000", 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.3974999999999997e-05 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834446+0000", 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2153000000000002e-05 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834469+0000", 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2762999999999999e-05 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.353 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.844909+0000", 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010440141 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.844932+0000", 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3033000000000001e-05 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.857464+0000", 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012531972000000001 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.354 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.857502+0000", 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8226000000000003e-05 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 88/33 ECSubWrite(tid=380, reqid=osd.11.0:380, at_version=88'255, trim_to=0'0, roll_forward_to=84'253))", 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.834943+0000", 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.45204603, 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011428252, 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.355 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834943+0000", 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834943+0000", 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.356 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.834943+0000", 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7700000000000004e-07 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835014+0000", 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1075999999999995e-05 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835044+0000", 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9745999999999999e-05 2024-09-16T04:02:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835065+0000", 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0483e-05 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835528+0000", 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00046368499999999998 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835554+0000", 2024-09-16T04:02:17.358 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6083000000000001e-05 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.846264+0000", 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010709847 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.846371+0000", 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000106755 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.359 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 88/33 ECSubWrite(tid=381, reqid=client.4646.0:904, at_version=88'81, trim_to=0'0, roll_forward_to=85'79))", 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.835122+0000", 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.4518668159999999, 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.023671623999999999, 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835122+0000", 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.360 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835122+0000", 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835123+0000", 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.4300000000000001e-07 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835173+0000", 2024-09-16T04:02:17.361 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0692999999999997e-05 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835197+0000", 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3180000000000002e-05 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835229+0000", 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2762000000000002e-05 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.836737+0000", 2024-09-16T04:02:17.362 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001507757 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.836753+0000", 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6178999999999999e-05 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.858713+0000", 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021959480999999999 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.363 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.858793+0000", 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0629000000000003e-05 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.cs3 88/33 ECSubWrite(tid=382, reqid=osd.11.0:382, at_version=88'257, trim_to=0'0, roll_forward_to=84'253))", 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.835274+0000", 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.451714832, 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021500160000000001, 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.364 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835274+0000", 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835274+0000", 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.835275+0000", 2024-09-16T04:02:17.365 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.7000000000000003e-07 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.836880+0000", 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0016047819999999999 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.836900+0000", 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0242999999999999e-05 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.366 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.836922+0000", 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1988999999999999e-05 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.837073+0000", 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000150851 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.837098+0000", 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5222999999999999e-05 2024-09-16T04:02:17.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.856676+0000", 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019577845999999999 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.856774+0000", 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8256000000000003e-05 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.368 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.9s6 88/33 ECSubWrite(tid=208, reqid=osd.14.0:208, at_version=88'37, trim_to=0'0, roll_forward_to=86'35))", 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.840343+0000", 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.4466453650000002, 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.017112624, 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.840343+0000", 2024-09-16T04:02:17.369 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.840343+0000", 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.840344+0000", 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6600000000000005e-07 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.370 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.840405+0000", 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.0429e-05 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.840437+0000", 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.256e-05 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.840460+0000", 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2898999999999999e-05 2024-09-16T04:02:17.371 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.844758+0000", 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042981310000000002 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.844781+0000", 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3003e-05 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.372 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.857362+0000", 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012580427 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.857456+0000", 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.4308999999999994e-05 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.373 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.9s6 88/33 ECSubWrite(tid=209, reqid=osd.14.0:209, at_version=88'39, trim_to=0'0, roll_forward_to=86'35))", 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.845055+0000", 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.4419334849999998, 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.016275839, 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.845055+0000", 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.374 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.845055+0000", 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.845057+0000", 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.103e-06 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.845153+0000", 2024-09-16T04:02:17.375 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5540999999999999e-05 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.845186+0000", 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3330000000000001e-05 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.845212+0000", 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6186e-05 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.847836+0000", 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0026239470000000002 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.847861+0000", 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4493000000000001e-05 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.861241+0000", 2024-09-16T04:02:17.377 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013379772 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.861331+0000", 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.0467e-05 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs5 88/33 ECSubWrite(tid=384, reqid=osd.11.0:384, at_version=88'85, trim_to=0'0, roll_forward_to=85'79))", 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:13.850315+0000", 2024-09-16T04:02:17.378 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 3.4366740949999999, 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014885252999999999, 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.850315+0000", 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.850315+0000", 2024-09-16T04:02:17.379 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.850316+0000", 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0529999999999999e-06 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.850390+0000", 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4091999999999995e-05 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.380 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.850421+0000", 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1513000000000001e-05 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.850442+0000", 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0373e-05 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.853207+0000", 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0027653909999999999 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.381 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.853232+0000", 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4943000000000002e-05 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.865104+0000", 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011871583 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:13.865200+0000", 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6304999999999994e-05 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.382 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.383 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.383 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.383 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.383 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:17.384 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:17.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:02:17.304+0000 7fc236d9d640 0 -- 172.21.15.37:0/661549869 >> [v2:172.21.15.188:6818/3000701187,v1:172.21.15.188:6819/3000701187] conn(0x7fc238226000 msgr2=0x7fc238228430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T04:02:17.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:02:17.386 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:17.380+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:17.486 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:17.484+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:17.531 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:17.532 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:17.550 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:17.551 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:17.551 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:02:17.551 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:17.564 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:02:17.587 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:17.584+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:17.687 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:17.687 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:17.687 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:02:17.687 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:02:17.687 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:17.688 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:17.684+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:17.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:02:17.732 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.7 primary-affinity to 0.057398 (837612) 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:368: done (2 left) 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:369: read oid 1 snap -1 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:369: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:370: read oid 23 snap -1 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:370: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:371: snap_create 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:365: finishing write tid 1 to smithi18831615-44 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:365: finishing write tid 2 to smithi18831615-44 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:367: finishing rollback tid 0 to smithi18831615-11 2024-09-16T04:02:17.740 INFO:tasks.rados.rados.0.smithi188.stdout:365: finishing write tid 3 to smithi18831615-44 2024-09-16T04:02:17.741 INFO:tasks.rados.rados.0.smithi188.stdout:365: oid 44 updating version 0 to 71 2024-09-16T04:02:17.741 INFO:tasks.rados.rados.0.smithi188.stdout:365: oid 44 updating version 71 to 72 2024-09-16T04:02:17.741 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 44 v 72 (ObjNum 147 snap 38 seq_num 147) dirty exists 2024-09-16T04:02:17.741 INFO:tasks.rados.rados.0.smithi188.stdout:365: left oid 44 (ObjNum 147 snap 38 seq_num 147) 2024-09-16T04:02:17.741 INFO:tasks.rados.rados.0.smithi188.stdout:367: finishing rollback tid 1 to smithi18831615-11 2024-09-16T04:02:17.741 INFO:tasks.rados.rados.0.smithi188.stdout:367: finishing rollback tid 2 to smithi18831615-11 2024-09-16T04:02:17.741 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 11 v 272 (ObjNum 127 snap 31 seq_num 127) dirty exists 2024-09-16T04:02:17.748 INFO:tasks.rados.rados.0.smithi188.stdout:369: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:02:17.758 INFO:tasks.rados.rados.0.smithi188.stdout:370: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:02:17.788 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:17.784+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:17.837 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:8 3.0s0 3:0076079f:::smithi18831615-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 55.988591059999997, 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.022018801000000001, 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.838 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 8 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845613+0000", 2024-09-16T04:02:17.839 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845615+0000", 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.967e-06 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845638+0000", 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2532e-05 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.840 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845663+0000", 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845680+0000", 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7323e-05 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.851526+0000", 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058460049999999996 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.841 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.851763+0000", 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023643399999999999 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.852890+0000", 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0011268949999999999 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.858042+0000", 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051525850000000003 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.842 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.867414+0000", 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0093716719999999993 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.867632+0000", 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00021833200000000001 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:9 3.0s0 3:0076079f:::smithi18831615-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T04:02:17.843 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 55.988495807, 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.027184505000000001, 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 9 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.844 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845709+0000", 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845710+0000", 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.567e-06 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.845 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845718+0000", 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7729999999999999e-06 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845740+0000", 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2048999999999998e-05 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.845757+0000", 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7076e-05 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.846 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.853731+0000", 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0079742270000000004 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.853981+0000", 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025017399999999998 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.868611+0000", 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014630026000000001 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.847 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.868826+0000", 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000214816 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.872893+0000", 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0040667969999999996 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:84 3.0s0 3:0baf0dec:::smithi18831615-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 55.930452164000002, 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021781656, 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 84 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.849 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903752+0000", 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903753+0000", 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1039999999999999e-06 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.850 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903759+0000", 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1380000000000004e-06 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903787+0000", 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7223999999999999e-05 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.851 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.903802+0000", 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5301000000000001e-05 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.908998+0000", 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0051958990000000004 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.909182+0000", 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000184143 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.852 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.920281+0000", 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.011099238000000001 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.920720+0000", 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00043861699999999998 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:21.925534+0000", 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0048139919999999996 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.853 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:327 3.0s0 3:03f9c209:::smithi18831615-40:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 50.944981486000003, 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.027429133000000001, 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 327 2024-09-16T04:02:17.854 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889223+0000", 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889227+0000", 2024-09-16T04:02:17.855 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8700000000000002e-06 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889241+0000", 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4606e-05 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889275+0000", 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3892e-05 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.856 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889293+0000", 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7353000000000001e-05 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889422+0000", 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000129799 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889435+0000", 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2167000000000001e-05 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.857 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.893120+0000", 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003685406 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.893360+0000", 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023993500000000001 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.895575+0000", 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0022147410000000001 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.858 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901769+0000", 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061945739999999996 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916453+0000", 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014683732999999999 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916652+0000", 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019905699999999999 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.859 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:328 3.0s0 3:03f9c209:::smithi18831615-40:head [rollback 3] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 50.944870530000003, 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.029259183000000001, 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.860 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 328 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889334+0000", 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.861 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889335+0000", 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6700000000000002e-07 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889340+0000", 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8230000000000002e-06 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889352+0000", 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.234e-05 2024-09-16T04:02:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889360+0000", 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889591+0000", 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023105599999999999 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889610+0000", 2024-09-16T04:02:17.863 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8749e-05 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.896810+0000", 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0072003049999999997 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.897074+0000", 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00026394400000000001 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.864 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.898389+0000", 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001314787 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916676+0000", 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018286922000000001 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918392+0000", 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001716038 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.865 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.918593+0000", 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000201289 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:329 3.0s0 3:03f9c209:::smithi18831615-40:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 50.944831106999999, 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.031812201999999998, 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.866 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 329 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889373+0000", 2024-09-16T04:02:17.867 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889374+0000", 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.0299999999999998e-07 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889377+0000", 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2770000000000001e-06 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.868 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889387+0000", 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.6260000000000007e-06 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889394+0000", 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9059999999999996e-06 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889709+0000", 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00031473299999999998 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.869 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889718+0000", 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2499999999999995e-06 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899428+0000", 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0097098859999999992 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.899700+0000", 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000272558 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.870 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.901104+0000", 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00140375 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920686+0000", 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019581923000000001 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.920950+0000", 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00026403800000000002 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.871 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921185+0000", 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00023545200000000001 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:330 3.0s0 3:0076079f:::smithi18831615-3:head [stat,delete] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 50.944765189000002, 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.032116411999999997, 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.872 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 330 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889439+0000", 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889440+0000", 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6000000000000003e-07 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889444+0000", 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.874 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889455+0000", 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0259e-05 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889462+0000", 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4329999999999998e-06 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889794+0000", 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000331966 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.875 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.889810+0000", 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5696999999999999e-05 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902422+0000", 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012612490000000001 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.902647+0000", 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00022501300000000001 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.876 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.906224+0000", 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0035769059999999999 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921192+0000", 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014967644 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921363+0000", 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000171304 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.877 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.921556+0000", 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019263700000000001 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s4 41/33 ECSubWrite(tid=10, reqid=osd.5.0:10, at_version=41'8, trim_to=0'0, roll_forward_to=37'6))", 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 50.930774174, 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.033655513999999997, 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.878 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903430+0000", 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.879 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903432+0000", 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2970000000000002e-06 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903515+0000", 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2059999999999997e-05 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903547+0000", 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2106000000000001e-05 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.880 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.903569+0000", 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1939000000000001e-05 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916774+0000", 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013205099 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.916797+0000", 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2809000000000001e-05 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.881 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937024+0000", 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020227256999999998 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:26.937086+0000", 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1946999999999995e-05 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:371 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:17.882 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 41.115925640999997, 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018868566, 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 371 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.883 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718279+0000", 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718281+0000", 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1969999999999999e-06 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.884 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718293+0000", 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1636e-05 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718315+0000", 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2935999999999999e-05 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718327+0000", 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1124000000000001e-05 2024-09-16T04:02:17.885 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718409+0000", 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2245999999999998e-05 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718416+0000", 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.657e-06 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.724610+0000", 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0061949750000000001 2024-09-16T04:02:17.886 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.724979+0000", 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00036804699999999998 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.727047+0000", 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0020681330000000002 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.887 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.735227+0000", 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0081803929999999993 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.737066+0000", 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00183858 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.737147+0000", 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1642000000000002e-05 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.888 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:372 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [rollback 7] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 41.115841291999999, 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021826534000000002, 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 372 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.889 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718363+0000", 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718364+0000", 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:02:17.890 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718367+0000", 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0489999999999999e-06 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718384+0000", 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7365000000000001e-05 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718398+0000", 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.358e-05 2024-09-16T04:02:17.891 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718493+0000", 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5422999999999998e-05 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718498+0000", 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.7500000000000003e-06 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.728402+0000", 2024-09-16T04:02:17.892 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0099045539999999994 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.728508+0000", 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000105475 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.729094+0000", 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00058619300000000004 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.893 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.736729+0000", 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0076349230000000001 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740059+0000", 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0033296939999999998 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740190+0000", 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00013097299999999999 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.894 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:373 3.0s0 3:0dbd8ad9:::smithi18831615-49:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 41.115793181999997, 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.022197585999999998, 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.895 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 373 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718411+0000", 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.896 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718412+0000", 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1099999999999995e-07 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718416+0000", 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.873e-06 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718427+0000", 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1233000000000001e-05 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718435+0000", 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7069999999999992e-06 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718521+0000", 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.6065999999999999e-05 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.898 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.718524+0000", 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.72e-06 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732081+0000", 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0135566 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732182+0000", 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000101362 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.899 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.732688+0000", 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00050549700000000004 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.736733+0000", 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.004045089 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740503+0000", 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003770088 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.900 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:36.740609+0000", 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010564 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:512 3.0s0 3:0baf0dec:::smithi18831615-28:head [copy-from ver 30] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:50.860592+0000", 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 26.973612827, 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.049257191999999998, 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.901 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 512 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860592+0000", 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.902 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860592+0000", 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860594+0000", 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6359999999999998e-06 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860608+0000", 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3366999999999999e-05 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.903 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860640+0000", 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2146e-05 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860657+0000", 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7523000000000001e-05 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860785+0000", 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012776700000000001 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.904 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860797+0000", 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.224e-05 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.864255+0000", 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003457781 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.864407+0000", 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015244999999999999 2024-09-16T04:02:17.905 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.901818+0000", 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.037410153000000002 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.908299+0000", 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0064817659999999999 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.909766+0000", 2024-09-16T04:02:17.906 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0014670900000000001 2024-09-16T04:02:17.907 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.907 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.907 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.907 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.909849+0000", 2024-09-16T04:02:17.907 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.2273000000000003e-05 2024-09-16T04:02:17.907 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.907 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.907 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:513 3.0s0 3:0baf0dec:::smithi18831615-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:50.860697+0000", 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 26.973507123000001, 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.047167210000000001, 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 513 2024-09-16T04:02:17.908 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860697+0000", 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860697+0000", 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.909 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860698+0000", 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5999999999999991e-07 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860704+0000", 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.4569999999999996e-06 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860716+0000", 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2823000000000001e-05 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.910 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860725+0000", 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.9230000000000008e-06 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860937+0000", 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00021142599999999999 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.860956+0000", 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.927e-05 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.911 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.864755+0000", 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037985340000000001 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.864922+0000", 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016783999999999999 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.907306+0000", 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.042383415000000001 2024-09-16T04:02:17.912 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.907559+0000", 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025296500000000001 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:50.907864+0000", 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00030559700000000001 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.913 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:559 3.0s0 3:0baf0dec:::smithi18831615-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:52.761892+0000", 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 25.072312147000002, 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020170703000000002, 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 559 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.914 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761892+0000", 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761892+0000", 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761893+0000", 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7300000000000005e-07 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.915 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761897+0000", 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.534e-06 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761907+0000", 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0317e-05 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.761914+0000", 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.3119999999999996e-06 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.916 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.765546+0000", 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0036314260000000001 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.765651+0000", 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010494599999999999 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.778221+0000", 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012570141 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.917 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.778348+0000", 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000127494 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:52.782063+0000", 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0037146599999999998 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.918 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:636 3.9s0 3:9300f872:::smithi18831615-22:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.919425+0000", 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 19.914779828, 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018013963000000001, 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.919 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 636 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919425+0000", 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919425+0000", 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.920 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919426+0000", 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7799999999999999e-06 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919435+0000", 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.7129999999999994e-06 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919456+0000", 2024-09-16T04:02:17.921 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1157000000000001e-05 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919471+0000", 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4876e-05 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919684+0000", 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000213148 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.922 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919701+0000", 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7197000000000002e-05 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.926948+0000", 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0072468239999999998 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.927206+0000", 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025767100000000001 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.923 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.937439+0000", 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010232597 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:637 3.9s0 3:9f17a056:::smithi18831615-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.919492+0000", 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 19.914711899, 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019324294999999998, 2024-09-16T04:02:17.924 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 637 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919492+0000", 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.925 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919492+0000", 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919494+0000", 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1400000000000001e-06 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.926 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919529+0000", 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5618999999999999e-05 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919549+0000", 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9565999999999999e-05 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919564+0000", 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.491e-05 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.927 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919815+0000", 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00025087699999999999 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919827+0000", 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2680000000000001e-05 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.928058+0000", 2024-09-16T04:02:17.928 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0082304679999999995 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.928223+0000", 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016547600000000001 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.938817+0000", 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010593559000000001 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.929 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:638 3.9s0 3:9f17a056:::smithi18831615-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.919585+0000", 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 19.914619144, 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.023803158000000001, 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 638 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.930 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919585+0000", 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919585+0000", 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919586+0000", 2024-09-16T04:02:17.931 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919595+0000", 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.3690000000000007e-06 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919613+0000", 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8147000000000001e-05 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.932 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919627+0000", 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.432e-05 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919938+0000", 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00031104499999999997 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.933 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919956+0000", 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7903000000000001e-05 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.938944+0000", 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.018987456999999999 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.939131+0000", 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00018752800000000001 2024-09-16T04:02:17.934 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.943388+0000", 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0042571960000000004 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:639 3.9s0 3:9f17a056:::smithi18831615-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:57.919667+0000", 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 19.914537760000002, 2024-09-16T04:02:17.935 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.024857047, 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 639 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.936 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919667+0000", 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919667+0000", 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919668+0000", 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2270000000000001e-06 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.937 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919675+0000", 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.5399999999999998e-06 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919695+0000", 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0106e-05 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.919711+0000", 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5265999999999999e-05 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.938 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.920054+0000", 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000343655 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.920071+0000", 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6852999999999999e-05 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.939 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.939740+0000", 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019668486999999998 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.940021+0000", 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00028088700000000001 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.940 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:57.944524+0000", 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0045030260000000003 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:767 3.0s0 3:03f9c209:::smithi18831615-40:head [rmxattr O\\Up[arf_O^_OTehidgQm\\_dWqiOgZeddoq[Yeim in=40b,rmxattr SXYNfcNts[adfhlcih^SbYXacRgpdjjt_UedQoso in=40b,rmxattr Y`URWapPpbRbcVnVY`QSNsZaZVYgbfr]kcmrp_g\\ in=40b,rmxattr fddhrQT_f\\R in=11b,rmxattr jffqjeh^Weeqcg^`^Wa]]on^og\\VRjnkZqT\\oRWq in=40b,rmxattr paYr[ki]Uj^gZYZXoSk]OhhpsdnXbWSQNQUhgNgp in=40b] snapc 1e=[1e,1d,1c,1b,19,18,17,16,15,14,13,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:02.732958+0000", 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.101246836, 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.033543912000000002, 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:17.941 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 767 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.732958+0000", 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.942 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.732958+0000", 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.732961+0000", 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7759999999999999e-06 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.732979+0000", 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7710000000000002e-05 2024-09-16T04:02:17.943 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.733020+0000", 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.1356000000000002e-05 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.733043+0000", 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.249e-05 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.944 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.733225+0000", 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00018210200000000001 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.733250+0000", 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4822999999999999e-05 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.742998+0000", 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0097484719999999993 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.945 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.743286+0000", 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00028776000000000002 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.746418+0000", 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.003131829 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.765674+0000", 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019256527999999998 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.946 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.766296+0000", 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00062195999999999996 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.766501+0000", 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00020510600000000001 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.947 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s4 77/33 ECSubWrite(tid=22, reqid=osd.5.0:22, at_version=77'17, trim_to=0'0, roll_forward_to=69'15))", 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:02.751868+0000", 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 15.082336398000001, 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019051899000000001, 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.751868+0000", 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.948 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.751868+0000", 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.751869+0000", 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.751919+0000", 2024-09-16T04:02:17.949 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.9892999999999998e-05 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.751950+0000", 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0528999999999999e-05 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.751969+0000", 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9015999999999999e-05 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.950 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.765597+0000", 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013627808999999999 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.765629+0000", 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2518999999999998e-05 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.770842+0000", 2024-09-16T04:02:17.951 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0052132080000000004 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:02.770920+0000", 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7495000000000003e-05 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:17.952 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:17.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:02:17.954 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:17.948+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:18.055 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:18.052+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:18.086 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:02:18.087 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:18.100 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:02:18.156 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:18.149+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:18.224 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:18.224 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:18.224 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:02:18.224 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:02:18.224 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:18.238 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:02:18.257 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:18.252+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:18.358 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:18.352+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:18.363 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:350 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 48.436077242000003, 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092273240000000003, 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.364 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 350 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923566+0000", 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923567+0000", 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7e-07 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923571+0000", 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2830000000000003e-06 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923578+0000", 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2810000000000003e-06 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923582+0000", 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1500000000000001e-06 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923849+0000", 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000266464 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923854+0000", 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.989e-06 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927191+0000", 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033374810000000002 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927250+0000", 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8369000000000003e-05 2024-09-16T04:02:18.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.932793+0000", 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055438370000000002 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.369 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:351 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:18.370 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:02:18.370 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 48.435994987999997, 2024-09-16T04:02:18.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097430910000000006, 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 351 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:02:18.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923648+0000", 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923649+0000", 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2399999999999999e-07 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923652+0000", 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.827e-06 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923659+0000", 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6420000000000001e-06 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923663+0000", 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7900000000000001e-06 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923914+0000", 2024-09-16T04:02:18.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000251387 2024-09-16T04:02:18.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:18.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.923926+0000", 2024-09-16T04:02:18.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1933000000000001e-05 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927543+0000", 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036171580000000001 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.927607+0000", 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3925999999999997e-05 2024-09-16T04:02:18.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:29.933391+0000", 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005784104 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.9s3 51/33 ECSubWrite(tid=75, reqid=client.4646.0:384, at_version=51'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.507125027999997, 2024-09-16T04:02:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099262940000000004, 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852518+0000", 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.852519+0000", 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000004e-07 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853510+0000", 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099128100000000011 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853546+0000", 2024-09-16T04:02:18.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5522000000000003e-05 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853572+0000", 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5924000000000001e-05 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853747+0000", 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017537299999999999 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.853774+0000", 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7129000000000001e-05 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.862371+0000", 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085965390000000003 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.862445+0000", 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3942000000000005e-05 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:620 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928440+0000", 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.431203234000002, 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020208812, 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.381 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 620 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928440+0000", 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928440+0000", 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928443+0000", 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.463e-06 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928454+0000", 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1259e-05 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928474+0000", 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0698000000000001e-05 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928490+0000", 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5546e-05 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928672+0000", 2024-09-16T04:02:18.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018192600000000001 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928682+0000", 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0492000000000001e-05 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.930911+0000", 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022290159999999999 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.931035+0000", 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000123149 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.948649+0000", 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017614263000000002 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:621 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:18.386 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928525+0000", 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.431117930999999, 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020965587000000001, 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 621 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928525+0000", 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928525+0000", 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928526+0000", 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3200000000000003e-07 2024-09-16T04:02:18.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928533+0000", 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928544+0000", 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0975000000000001e-05 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928553+0000", 2024-09-16T04:02:18.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9050000000000003e-06 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928920+0000", 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036759500000000002 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928947+0000", 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6876000000000002e-05 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.940657+0000", 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011709913000000001 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.940806+0000", 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014908599999999999 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.949491+0000", 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086846979999999994 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:622 3.as0 3:57198856:::smithi18831615-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928566+0000", 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.431076892, 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021638297000000001, 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.392 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 622 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928566+0000", 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928566+0000", 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928567+0000", 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4600000000000004e-07 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928573+0000", 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6799999999999998e-06 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928583+0000", 2024-09-16T04:02:18.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7389999999999994e-06 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928593+0000", 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9299999999999998e-06 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.929017+0000", 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042487699999999999 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:18.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.929028+0000", 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0302000000000001e-05 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941680+0000", 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012652029 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941790+0000", 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110104 2024-09-16T04:02:18.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.950205+0000", 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0084148899999999995 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpReadReply(3.as0 70/33 ECSubReadReply(tid=20, attrs_read=0))", 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.932357+0000", 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.427286177999999, 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0088944809999999992, 2024-09-16T04:02:18.397 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.932357+0000", 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.932357+0000", 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.932358+0000", 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0189999999999999e-06 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.934001+0000", 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00164285 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.934024+0000", 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3436000000000001e-05 2024-09-16T04:02:18.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.934041+0000", 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6909e-05 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941200+0000", 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071581789999999998 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941252+0000", 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2088e-05 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 81/33 ECSubWrite(tid=281, reqid=osd.13.0:281, at_version=81'59, trim_to=0'0, roll_forward_to=77'57))", 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:06.746627+0000", 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.61301667, 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010450789, 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.401 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746627+0000", 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746627+0000", 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746627+0000", 2024-09-16T04:02:18.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4300000000000003e-07 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746685+0000", 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8148e-05 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746712+0000", 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6202000000000001e-05 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746736+0000", 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4073000000000001e-05 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746927+0000", 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019125199999999999 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746948+0000", 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1165000000000001e-05 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757008+0000", 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010059742 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757077+0000", 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9664000000000003e-05 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.405 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 81/33 ECSubWrite(tid=282, reqid=osd.13.0:282, at_version=81'61, trim_to=0'0, roll_forward_to=77'57))", 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:06.746793+0000", 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.612849916, 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018759426999999999, 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746793+0000", 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746793+0000", 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746794+0000", 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7599999999999996e-07 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746841+0000", 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6777999999999998e-05 2024-09-16T04:02:18.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746863+0000", 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2116e-05 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.746880+0000", 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7136e-05 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.752354+0000", 2024-09-16T04:02:18.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054740889999999997 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.752392+0000", 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7798e-05 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.765463+0000", 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013070544 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.765553+0000", 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0190000000000002e-05 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 81/33 ECSubWrite(tid=91, reqid=osd.7.0:91, at_version=81'69, trim_to=0'0, roll_forward_to=79'67))", 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:06.754787+0000", 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.604856441999999, 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011902769000000001, 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.410 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.754787+0000", 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.754787+0000", 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.754788+0000", 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4700000000000001e-07 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.754840+0000", 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2688000000000001e-05 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.754885+0000", 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4920999999999999e-05 2024-09-16T04:02:18.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.754907+0000", 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1739e-05 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757848+0000", 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029407090000000001 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757875+0000", 2024-09-16T04:02:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6993000000000001e-05 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.766593+0000", 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087185189999999992 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.766690+0000", 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6552999999999995e-05 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.414 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 81/33 ECSubWrite(tid=92, reqid=osd.7.0:92, at_version=81'71, trim_to=0'0, roll_forward_to=79'67))", 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:06.757677+0000", 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.601966278000001, 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010203465, 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757677+0000", 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757677+0000", 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757678+0000", 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4529999999999999e-06 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757810+0000", 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013178500000000001 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.759478+0000", 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001667902 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.759509+0000", 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1136000000000001e-05 2024-09-16T04:02:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.760133+0000", 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062403199999999995 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.760150+0000", 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6948000000000001e-05 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.767789+0000", 2024-09-16T04:02:18.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007638873 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.767880+0000", 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1335999999999994e-05 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 82/33 ECSubWrite(tid=96, reqid=client.4646.0:829, at_version=82'77, trim_to=0'0, roll_forward_to=81'71))", 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.796110+0000", 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.563533015000001, 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010284158999999999, 2024-09-16T04:02:18.419 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.796110+0000", 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.796110+0000", 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.796111+0000", 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8899999999999998e-07 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.798734+0000", 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026234019999999999 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.798750+0000", 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5770999999999999e-05 2024-09-16T04:02:18.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.798761+0000", 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0659000000000001e-05 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.798881+0000", 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000119641 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.798905+0000", 2024-09-16T04:02:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4182e-05 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.806335+0000", 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074305170000000002 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.806394+0000", 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9197999999999998e-05 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.423 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:897 3.as0 3:57198856:::smithi18831615-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:11.867502+0000", 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.4921409509999997, 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092708189999999996, 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.424 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 897 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.867502+0000", 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.867502+0000", 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.867503+0000", 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.79e-07 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.867505+0000", 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8169999999999999e-06 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.867509+0000", 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0430000000000001e-06 2024-09-16T04:02:18.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.867512+0000", 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9909999999999998e-06 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.869782+0000", 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022693290000000001 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.869873+0000", 2024-09-16T04:02:18.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1458000000000004e-05 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.873892+0000", 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040190299999999998 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.873971+0000", 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8770999999999995e-05 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:11.876773+0000", 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002802101 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 88/33 ECSubWrite(tid=300, reqid=osd.13.0:300, at_version=88'65, trim_to=0'0, roll_forward_to=82'63))", 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:13.828917+0000", 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5307266400000001, 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011755648, 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.429 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.828917+0000", 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.828917+0000", 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.828917+0000", 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3000000000000001e-07 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.828954+0000", 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7239e-05 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.828985+0000", 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0929000000000002e-05 2024-09-16T04:02:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829010+0000", 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4581999999999999e-05 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829225+0000", 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021494999999999999 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829250+0000", 2024-09-16T04:02:18.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5649000000000001e-05 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.840254+0000", 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011003811 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.840672+0000", 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041805800000000001 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 88/33 ECSubWrite(tid=116, reqid=osd.7.0:116, at_version=88'95, trim_to=0'0, roll_forward_to=86'93))", 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:13.830441+0000", 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5292026830000003, 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010203703, 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830441+0000", 2024-09-16T04:02:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830441+0000", 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830442+0000", 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9300000000000006e-07 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830501+0000", 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9262000000000003e-05 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830537+0000", 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5692000000000003e-05 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830560+0000", 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3872000000000001e-05 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830758+0000", 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019762 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830784+0000", 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5812999999999998e-05 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.840222+0000", 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0094385879999999995 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.840644+0000", 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042186300000000002 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 88/33 ECSubWrite(tid=117, reqid=osd.7.0:117, at_version=88'97, trim_to=0'0, roll_forward_to=86'93))", 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:13.830640+0000", 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5290030530000003, 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014279170000000001, 2024-09-16T04:02:18.438 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830640+0000", 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830640+0000", 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830641+0000", 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1370000000000001e-06 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830692+0000", 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0900999999999999e-05 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830718+0000", 2024-09-16T04:02:18.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5856000000000001e-05 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.830737+0000", 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9222999999999999e-05 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831804+0000", 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010662619999999999 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831828+0000", 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4272000000000001e-05 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.844811+0000", 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012983491999999999 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.844919+0000", 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108027 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.442 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s2 88/33 ECSubWrite(tid=301, reqid=osd.13.0:301, at_version=88'48, trim_to=0'0, roll_forward_to=86'46))", 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:13.831224+0000", 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5284189770000003, 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019902476999999998, 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831224+0000", 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831224+0000", 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831226+0000", 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7540000000000001e-06 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831279+0000", 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2929000000000001e-05 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831309+0000", 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0048000000000001e-05 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831331+0000", 2024-09-16T04:02:18.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1699e-05 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831514+0000", 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000183609 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831537+0000", 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2976000000000002e-05 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.851051+0000", 2024-09-16T04:02:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019513875 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.851127+0000", 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5587000000000007e-05 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 88/33 ECSubWrite(tid=302, reqid=osd.13.0:302, at_version=88'67, trim_to=0'0, roll_forward_to=82'63))", 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:13.833082+0000", 2024-09-16T04:02:18.447 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5265615639999996, 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011927218, 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833082+0000", 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833082+0000", 2024-09-16T04:02:18.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833084+0000", 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.792e-06 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833131+0000", 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7867000000000001e-05 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833160+0000", 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8943999999999999e-05 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833187+0000", 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.69e-05 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833350+0000", 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016295 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833376+0000", 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5332000000000001e-05 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.844945+0000", 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011569525000000001 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.845009+0000", 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3907999999999999e-05 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.451 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.2s2 88/33 ECSubWrite(tid=303, reqid=osd.13.0:301, at_version=0'0, trim_to=0'0, roll_forward_to=88'48))", 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:13.852725+0000", 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5069180549999999, 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010309979, 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.852725+0000", 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.852725+0000", 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.852727+0000", 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6929999999999999e-06 2024-09-16T04:02:18.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.852757+0000", 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0148999999999999e-05 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.852788+0000", 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0959e-05 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.852809+0000", 2024-09-16T04:02:18.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1285999999999999e-05 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.853110+0000", 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000300177 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.853137+0000", 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7124999999999999e-05 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.862973+0000", 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.009835926 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.863035+0000", 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2663999999999995e-05 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s6 88/33 ECSubWrite(tid=304, reqid=osd.13.0:302, at_version=0'0, trim_to=0'0, roll_forward_to=88'67))", 2024-09-16T04:02:18.456 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:13.854307+0000", 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.5053366740000005, 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0084204510000000007, 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.854307+0000", 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.854307+0000", 2024-09-16T04:02:18.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.854307+0000", 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4e-07 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.854346+0000", 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8936000000000001e-05 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.854371+0000", 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4681999999999998e-05 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.854391+0000", 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9732999999999999e-05 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.854538+0000", 2024-09-16T04:02:18.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014669800000000001 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.854563+0000", 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5899999999999999e-05 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.862639+0000", 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080756519999999991 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.862727+0000", 2024-09-16T04:02:18.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8010000000000006e-05 2024-09-16T04:02:18.461 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.461 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.461 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.461 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.461 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.461 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:18.462 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:02:18.463 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:18.457+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:18.563 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:18.560+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:18.633 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:02:18.633 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:18.648 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:02:18.664 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:18.660+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:18.741 INFO:tasks.rados.rados.0.smithi188.stdout:365: done (4 left) 2024-09-16T04:02:18.741 INFO:tasks.rados.rados.0.smithi188.stdout:367: done (3 left) 2024-09-16T04:02:18.741 INFO:tasks.rados.rados.0.smithi188.stdout:369: done (2 left) 2024-09-16T04:02:18.741 INFO:tasks.rados.rados.0.smithi188.stdout:370: done (1 left) 2024-09-16T04:02:18.741 INFO:tasks.rados.rados.0.smithi188.stdout:371: done (0 left) 2024-09-16T04:02:18.741 INFO:tasks.rados.rados.0.smithi188.stdout:372: copy_from oid 11 from oid 21 current snap is 39 2024-09-16T04:02:18.742 INFO:tasks.rados.rados.0.smithi188.stdout:373: delete oid 39 current snap is 39 2024-09-16T04:02:18.754 INFO:tasks.rados.rados.0.smithi188.stdout:373: done (1 left) 2024-09-16T04:02:18.755 INFO:tasks.rados.rados.0.smithi188.stdout:374: read oid 6 snap -1 2024-09-16T04:02:18.755 INFO:tasks.rados.rados.0.smithi188.stdout:374: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T04:02:18.755 INFO:tasks.rados.rados.0.smithi188.stdout:375: rmattr oid 22 current snap is 39 2024-09-16T04:02:18.755 INFO:tasks.rados.rados.0.smithi188.stdout:375: done (2 left) 2024-09-16T04:02:18.755 INFO:tasks.rados.rados.0.smithi188.stdout:376: setattr oid 35 current snap is 39 2024-09-16T04:02:18.765 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:02:18.766 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:18.760+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:18.767 INFO:tasks.rados.rados.0.smithi188.stdout:372: finishing copy_from racing read to smithi18831615-11 2024-09-16T04:02:18.768 INFO:tasks.rados.rados.0.smithi188.stdout:377: copy_from oid 43 from oid 18 current snap is 39 2024-09-16T04:02:18.768 INFO:tasks.rados.rados.0.smithi188.stdout:378: delete oid 25 current snap is 39 2024-09-16T04:02:18.773 INFO:tasks.rados.rados.0.smithi188.stdout:374: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T04:02:18.778 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:18.779 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:18.779 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:02:18.779 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:02:18.779 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:18.796 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:02:18.844 INFO:tasks.rados.rados.0.smithi188.stdout:372: finishing copy_from to smithi18831615-11 2024-09-16T04:02:18.844 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 11 v 152 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-09-16T04:02:18.844 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 35 v 109 (ObjNum 90 snap 16 seq_num 90) dirty exists 2024-09-16T04:02:18.844 INFO:tasks.rados.rados.0.smithi188.stdout:377: finishing copy_from racing read to smithi18831615-43 2024-09-16T04:02:18.844 INFO:tasks.rados.rados.0.smithi188.stdout:377: finishing copy_from to smithi18831615-43 2024-09-16T04:02:18.844 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 43 v 16 (ObjNum 76 snap 3 seq_num 76) dirty exists 2024-09-16T04:02:18.845 INFO:tasks.rados.rados.0.smithi188.stdout:372: done (4 left) 2024-09-16T04:02:18.845 INFO:tasks.rados.rados.0.smithi188.stdout:374: done (3 left) 2024-09-16T04:02:18.845 INFO:tasks.rados.rados.0.smithi188.stdout:376: done (2 left) 2024-09-16T04:02:18.845 INFO:tasks.rados.rados.0.smithi188.stdout:377: done (1 left) 2024-09-16T04:02:18.845 INFO:tasks.rados.rados.0.smithi188.stdout:378: done (0 left) 2024-09-16T04:02:18.845 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 7 current snap is 39 2024-09-16T04:02:18.845 INFO:tasks.rados.rados.0.smithi188.stdout:379: seq_num 150 ranges {2342912=114688} 2024-09-16T04:02:18.865 INFO:tasks.rados.rados.0.smithi188.stdout:379: writing smithi18831615-7 from 2342912 to 2457600 tid 1 2024-09-16T04:02:18.865 INFO:tasks.rados.rados.0.smithi188.stdout:380: snap_remove snap 33 2024-09-16T04:02:18.865 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:18.860+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:18.962 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:18.962 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:419 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~671744] snapc c=[c,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.167355301999997, 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019325796999999999, 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.963 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 419 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792508+0000", 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.792510+0000", 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5359999999999999e-06 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793316+0000", 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00080598599999999996 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793351+0000", 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4473e-05 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793368+0000", 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6943e-05 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793504+0000", 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013629000000000001 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.793707+0000", 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020315399999999999 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.796655+0000", 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002948301 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.800082+0000", 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003426999 2024-09-16T04:02:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.811562+0000", 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011479896999999999 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.811833+0000", 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027121799999999999 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:422 3.ds0 3:bab6d723:::smithi18831615-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.152837173000002, 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02126656, 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 422 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.969 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807026+0000", 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807027+0000", 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3969999999999999e-06 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807034+0000", 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1459999999999999e-06 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807055+0000", 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.09e-05 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.807070+0000", 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4932999999999999e-05 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814329+0000", 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007259138 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.814519+0000", 2024-09-16T04:02:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000189435 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.822102+0000", 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075835199999999998 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.822353+0000", 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025044999999999999 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:39.828292+0000", 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059396409999999998 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:427 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:02:18.974 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.237665722000003, 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018266843000000001, 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 427 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:02:18.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722197+0000", 2024-09-16T04:02:18.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722200+0000", 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6929999999999999e-06 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722212+0000", 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2577e-05 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722245+0000", 2024-09-16T04:02:18.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2162999999999997e-05 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722262+0000", 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6843000000000001e-05 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722406+0000", 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000144668 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:18.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722421+0000", 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5271999999999999e-05 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.728611+0000", 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006189193 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.728887+0000", 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027599500000000002 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.740464+0000", 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011577439 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:428 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.237563710000003, 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019155492999999999, 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.980 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 428 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722299+0000", 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722300+0000", 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7999999999999993e-07 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722308+0000", 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8259999999999993e-06 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722350+0000", 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1730000000000002e-05 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722359+0000", 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5289999999999997e-06 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722581+0000", 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000221623 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722605+0000", 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4116000000000001e-05 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730680+0000", 2024-09-16T04:02:18.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080751769999999994 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.730940+0000", 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000259382 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741455+0000", 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010515129999999999 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:429 3.ds0 3:b60aff58:::smithi18831615-34:7 [read 0~720896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.237486844999999, 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024602057, 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 429 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.986 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722376+0000", 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722377+0000", 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722382+0000", 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7770000000000002e-06 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722392+0000", 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0020000000000001e-05 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722399+0000", 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4000000000000003e-06 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722655+0000", 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025568500000000002 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.722664+0000", 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4129999999999995e-06 2024-09-16T04:02:18.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741688+0000", 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019023476000000001 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.741991+0000", 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030357799999999999 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:41.746978+0000", 2024-09-16T04:02:18.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049869179999999999 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:521 3.ds0 3:b4418190:::smithi18831615-35:head [read 0~1687552,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861690+0000", 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.098173211999999, 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022625292000000002, 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.991 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 521 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861690+0000", 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861690+0000", 2024-09-16T04:02:18.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861691+0000", 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.753e-06 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861702+0000", 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0451999999999999e-05 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861729+0000", 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6988999999999999e-05 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861746+0000", 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7176e-05 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.862087+0000", 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000340584 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.862110+0000", 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3281999999999998e-05 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.869108+0000", 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0069983629999999996 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.869323+0000", 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021468100000000001 2024-09-16T04:02:18.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.884315+0000", 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992012000000001 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:522 3.ds0 3:b4418190:::smithi18831615-35:head [read 0~1687552,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861797+0000", 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.098065708, 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027488874999999999, 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 522 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:18.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861797+0000", 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861797+0000", 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861799+0000", 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.44e-06 2024-09-16T04:02:18.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861809+0000", 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0424999999999999e-05 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861831+0000", 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2345999999999999e-05 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:18.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861847+0000", 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5461999999999999e-05 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.862212+0000", 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036481699999999997 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.862233+0000", 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1257999999999999e-05 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.871538+0000", 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093055050000000004 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.871820+0000", 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000281351 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.889286+0000", 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017466270999999998 2024-09-16T04:02:19.001 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(osd.11.22:2 3.ds0 3:bab6d723:::smithi18831615-7:head [assert-version v51,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.009126+0000", 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.950736651, 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030180828, 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009126+0000", 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009126+0000", 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009129+0000", 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9830000000000001e-06 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009141+0000", 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.163e-05 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009162+0000", 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1299000000000001e-05 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009177+0000", 2024-09-16T04:02:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5031999999999999e-05 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009257+0000", 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0247999999999995e-05 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.009338+0000", 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0375000000000002e-05 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.039307+0000", 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029969261000000001 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:761 3.ds0 3:b4418190:::smithi18831615-35:head [copy-from ver 27] snapc 1e=[1e,1d,1c,1b,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.942736+0000", 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.017126705999999, 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028665875, 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 761 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942736+0000", 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942736+0000", 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942739+0000", 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2900000000000001e-06 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942749+0000", 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0920999999999999e-05 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942776+0000", 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6097000000000001e-05 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942787+0000", 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1676e-05 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942929+0000", 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014148899999999999 2024-09-16T04:02:19.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.943063+0000", 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000134454 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.962776+0000", 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019712398999999999 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.967865+0000", 2024-09-16T04:02:19.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050889569999999999 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.971298+0000", 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00343298 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.971402+0000", 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010461199999999999 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:762 3.ds0 3:b4418190:::smithi18831615-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.942824+0000", 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.017038862, 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02255515, 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 762 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.012 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942824+0000", 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942824+0000", 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942825+0000", 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-06 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942830+0000", 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9200000000000003e-06 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942848+0000", 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7453e-05 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.942866+0000", 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8603000000000001e-05 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.943371+0000", 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050457099999999999 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.943466+0000", 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5042000000000002e-05 2024-09-16T04:02:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.965096+0000", 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021629808 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.965233+0000", 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013705400000000001 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.965379+0000", 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014663299999999999 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/33 ECSubWrite(tid=56, reqid=osd.15.0:56, at_version=81'47, trim_to=0'0, roll_forward_to=77'45))", 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:06.755456+0000", 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.204407081999999, 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018963656999999998, 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.755456+0000", 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.755456+0000", 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.755457+0000", 2024-09-16T04:02:19.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5999999999999991e-07 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.755501+0000", 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3684999999999999e-05 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.755528+0000", 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7985000000000001e-05 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.755550+0000", 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1406000000000001e-05 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757506+0000", 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019563390000000001 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.757525+0000", 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8359000000000002e-05 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.774261+0000", 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016736232 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.774420+0000", 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000158691 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/33 ECSubWrite(tid=57, reqid=osd.15.0:57, at_version=81'49, trim_to=0'0, roll_forward_to=77'45))", 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:06.758386+0000", 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.201477265999999, 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024643327, 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.758386+0000", 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.758386+0000", 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.758387+0000", 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.81e-06 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.758461+0000", 2024-09-16T04:02:19.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3297e-05 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.758508+0000", 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6961e-05 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.758544+0000", 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5799000000000001e-05 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.761167+0000", 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026237080000000002 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.761197+0000", 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0182999999999999e-05 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.782965+0000", 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021767479999999999 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:06.783029+0000", 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4089000000000003e-05 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:829 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~671744] snapc 21=[21,20,1f,1e,1d,1c,1b,19,18,17,16,15,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.785840+0000", 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.174022829, 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033678590000000001, 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.026 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 829 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.785840+0000", 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.785840+0000", 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.785844+0000", 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8639999999999998e-06 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.786884+0000", 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010401010000000001 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.786916+0000", 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1785000000000001e-05 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.786938+0000", 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1818999999999999e-05 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.792620+0000", 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0056819289999999996 2024-09-16T04:02:19.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.792899+0000", 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027925299999999998 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.797130+0000", 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042308570000000002 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.803715+0000", 2024-09-16T04:02:19.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065857349999999997 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.819313+0000", 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015597705999999999 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.819519+0000", 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000205541 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 82/33 ECSubWrite(tid=59, reqid=osd.15.0:59, at_version=82'51, trim_to=0'0, roll_forward_to=81'49))", 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.788122+0000", 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.171740721999999, 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.037069617999999999, 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788122+0000", 2024-09-16T04:02:19.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788122+0000", 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788123+0000", 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9299999999999999e-07 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788157+0000", 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4208999999999997e-05 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788189+0000", 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2166999999999999e-05 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788201+0000", 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2224e-05 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.798845+0000", 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010644206 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.798871+0000", 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5792999999999999e-05 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.825148+0000", 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026276622999999999 2024-09-16T04:02:19.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.825192+0000", 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4103000000000001e-05 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.5s4 82/33 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=82'53, trim_to=0'0, roll_forward_to=81'49))", 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.788223+0000", 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.171640087, 2024-09-16T04:02:19.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.037000459999999999, 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788223+0000", 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788223+0000", 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788223+0000", 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2e-07 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788250+0000", 2024-09-16T04:02:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6979e-05 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788262+0000", 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1685000000000001e-05 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.788270+0000", 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4230000000000005e-06 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.801740+0000", 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013469462 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.801770+0000", 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9932000000000001e-05 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.825197+0000", 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023426929999999999 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.825223+0000", 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6628999999999999e-05 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:830 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~573440] snapc 21=[21,20,1f,1e,1d,1c,1b,19,18,17,16,15,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.792097+0000", 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.167765944999999, 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033793059, 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.041 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 830 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.792097+0000", 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.792097+0000", 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.792100+0000", 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.14e-06 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.792926+0000", 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00082579599999999995 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.792954+0000", 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7736e-05 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.792973+0000", 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9575000000000001e-05 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.803802+0000", 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010828426 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.807477+0000", 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003674877 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.809364+0000", 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018873060000000001 2024-09-16T04:02:19.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.825556+0000", 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016192112000000002 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.825763+0000", 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020746299999999999 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.825890+0000", 2024-09-16T04:02:19.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012662799999999999 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:831 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~753664] snapc 21=[21,20,1f,1e,1d,1c,1b,19,18,17,16,15,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.800683+0000", 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.159179634999999, 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026085312999999999, 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.047 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 831 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.800683+0000", 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.800683+0000", 2024-09-16T04:02:19.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.800686+0000", 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1659999999999998e-06 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.801735+0000", 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010483179999999999 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.801765+0000", 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0582e-05 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.801785+0000", 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9153000000000001e-05 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.811258+0000", 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0094734260000000001 2024-09-16T04:02:19.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.811533+0000", 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027464600000000003 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.815138+0000", 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036048970000000001 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.826111+0000", 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010973073 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.826381+0000", 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027085199999999997 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.826769+0000", 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038719999999999998 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.052 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:832 3.ds0 3:bab6d723:::smithi18831615-7:head [append 0~229376] snapc 21=[21,20,1f,1e,1d,1c,1b,19,18,17,16,15,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.802116+0000", 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.157747406, 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028648232999999999, 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.053 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 832 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802116+0000", 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802116+0000", 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802118+0000", 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1399999999999998e-06 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802520+0000", 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00040249200000000001 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802545+0000", 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4814999999999999e-05 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802563+0000", 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8162999999999999e-05 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.820265+0000", 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017701503 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.820521+0000", 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025603099999999999 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.822840+0000", 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023191090000000002 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.826113+0000", 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032728459999999998 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.830635+0000", 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045225789999999997 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.830764+0000", 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000128555 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:833 3.ds0 3:bab6d723:::smithi18831615-7:head [setxattr _header (56) in=7b] snapc 21=[21,20,1f,1e,1d,1c,1b,19,18,17,16,15,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.802593+0000", 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.157269704000001, 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030915970000000001, 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 833 2024-09-16T04:02:19.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802593+0000", 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802593+0000", 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802595+0000", 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.604e-06 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802603+0000", 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9959999999999993e-06 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802621+0000", 2024-09-16T04:02:19.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8529000000000002e-05 2024-09-16T04:02:19.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802636+0000", 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4797000000000001e-05 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.823860+0000", 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021223565 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.824012+0000", 2024-09-16T04:02:19.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000152676 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.824861+0000", 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084889700000000004 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.827265+0000", 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002404177 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.833390+0000", 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061248869999999999 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.833509+0000", 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118842 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:834 3.ds0 3:bab6d723:::smithi18831615-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:02:19.066 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:07.802654+0000", 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.157208755999999, 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033541148, 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 834 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802654+0000", 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802654+0000", 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802656+0000", 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3769999999999999e-06 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802661+0000", 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3759999999999999e-06 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802679+0000", 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8396000000000001e-05 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.802698+0000", 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9096000000000001e-05 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.827033+0000", 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024334208999999999 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.827239+0000", 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020639500000000001 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.834197+0000", 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0069580909999999996 2024-09-16T04:02:19.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.834360+0000", 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016318300000000001 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:07.836195+0000", 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018350249999999999 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.071 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.072 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:19.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:02:19.074 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:19.068+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:19.174 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:19.172+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:19.243 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:19.243 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:19.243 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:02:19.243 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:19.257 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:02:19.275 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:19.272+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:19.378 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:19.372+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:19.379 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:19.380 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:19.380 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:02:19.380 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:02:19.380 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:19.394 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:02:19.478 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:19.477+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:19.550 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:02:19.560 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:301 3.cs0 3:3922c872:::smithi18831615-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 53.644431001999997, 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024704912999999998, 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.561 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 301 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911957+0000", 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911958+0000", 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911962+0000", 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.63e-06 2024-09-16T04:02:19.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911976+0000", 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4027e-05 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.911985+0000", 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7530000000000007e-06 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928563+0000", 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016578809 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.928680+0000", 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117123 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931348+0000", 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002667141 2024-09-16T04:02:19.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.931647+0000", 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029889700000000001 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:25.936662+0000", 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050156630000000001 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:366 3.cs0 3:389a65d8:::smithi18831615-2:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.837439605, 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022413889999999999, 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 366 2024-09-16T04:02:19.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718949+0000", 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718950+0000", 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.663e-06 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718960+0000", 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.893e-06 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718982+0000", 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1820000000000001e-05 2024-09-16T04:02:19.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.718999+0000", 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7280000000000001e-05 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719156+0000", 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015617699999999999 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:19.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719166+0000", 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0832999999999999e-05 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.730920+0000", 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01175319 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.731179+0000", 2024-09-16T04:02:19.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000259588 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.741363+0000", 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010183446000000001 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:367 3.cs0 3:389a65d8:::smithi18831615-2:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:02:19.572 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.837334194, 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024825495, 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 367 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719054+0000", 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719056+0000", 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4839999999999999e-06 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719063+0000", 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7959999999999996e-06 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719085+0000", 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1365999999999998e-05 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719100+0000", 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4863e-05 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719209+0000", 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108978 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:19.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.719218+0000", 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1339999999999993e-06 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733259+0000", 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014041420000000001 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.733873+0000", 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061368800000000004 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.743880+0000", 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010006766 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.578 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:396 3.cs0 3:3631f100:::smithi18831615-24:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.683024746000001, 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024842948, 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 396 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873364+0000", 2024-09-16T04:02:19.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.01e-07 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873368+0000", 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.092e-06 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873374+0000", 2024-09-16T04:02:19.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5949999999999998e-06 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873378+0000", 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0949999999999998e-06 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.886123+0000", 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012745645999999999 2024-09-16T04:02:19.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.886245+0000", 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012146699999999999 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.898206+0000", 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011961652 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:401 3.cs0 3:3ac353cc:::smithi18831615-37:head [rollback 7] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.682898475000002, 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02229078, 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 401 2024-09-16T04:02:19.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873490+0000", 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1799999999999995e-07 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873492+0000", 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.657e-06 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873499+0000", 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.131e-06 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873504+0000", 2024-09-16T04:02:19.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.305e-06 2024-09-16T04:02:19.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888149+0000", 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014644292999999999 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888207+0000", 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8229000000000001e-05 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.888580+0000", 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037271 2024-09-16T04:02:19.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.891242+0000", 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026624280000000001 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895627+0000", 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043851180000000003 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895781+0000", 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015339100000000001 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.590 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:402 3.cs0 3:3ac353cc:::smithi18831615-37:head [append 0~0] snapc b=[a,9,8,7,5] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.682878332000001, 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022474063999999998, 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 402 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873510+0000", 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0499999999999999e-07 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873531+0000", 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0361e-05 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873538+0000", 2024-09-16T04:02:19.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2219999999999996e-06 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.873543+0000", 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8189999999999999e-06 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889058+0000", 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015515467 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889129+0000", 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0946999999999997e-05 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.889436+0000", 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030638799999999999 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.891244+0000", 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001808289 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895920+0000", 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046766380000000003 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:36.895984+0000", 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3528000000000006e-05 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.596 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:475 3.cs0 3:3f3e33cc:::smithi18831615-38:head [setxattr _header (57) in=7b] snapc 12=[12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:46.742977+0000", 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.813411559000002, 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025134200999999998, 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 475 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742977+0000", 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742977+0000", 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742978+0000", 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0759999999999999e-06 2024-09-16T04:02:19.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742984+0000", 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7440000000000004e-06 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.742995+0000", 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1686e-05 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743004+0000", 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3639999999999999e-06 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.757412+0000", 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014408746 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.757586+0000", 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017383200000000001 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.758544+0000", 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00095736499999999995 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.761119+0000", 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025749779999999999 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.767922+0000", 2024-09-16T04:02:19.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068035869999999998 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.768111+0000", 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018882299999999999 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:476 3.cs0 3:3f3e33cc:::smithi18831615-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:46.743016+0000", 2024-09-16T04:02:19.602 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.813372065999999, 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028572347000000001, 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 476 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743016+0000", 2024-09-16T04:02:19.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743016+0000", 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743017+0000", 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7300000000000005e-07 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743021+0000", 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8099999999999999e-06 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743031+0000", 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0665999999999999e-05 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.743041+0000", 2024-09-16T04:02:19.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0006999999999999e-05 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.759305+0000", 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016263283999999999 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.759504+0000", 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019926000000000001 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.768977+0000", 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0094726040000000008 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.769120+0000", 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000143022 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:46.771589+0000", 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024689209999999998 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.607 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:514 3.cs0 3:35c140c5:::smithi18831615-48:head [read 0~2211840,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861354+0000", 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.695034434, 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022129296999999999, 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.608 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 514 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861354+0000", 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861354+0000", 2024-09-16T04:02:19.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861358+0000", 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4730000000000002e-06 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861374+0000", 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5818999999999999e-05 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861425+0000", 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1047000000000001e-05 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861451+0000", 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5758999999999999e-05 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861658+0000", 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020674800000000001 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861678+0000", 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0522000000000001e-05 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867064+0000", 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053854799999999998 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867293+0000", 2024-09-16T04:02:19.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022948700000000001 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.883483+0000", 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016189961999999999 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:515 3.cs0 3:35c140c5:::smithi18831615-48:head [read 0~2211840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:19.613 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861550+0000", 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.694838612000002, 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025758455, 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 515 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861550+0000", 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861550+0000", 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861551+0000", 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.61e-06 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861562+0000", 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0633e-05 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861602+0000", 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9660999999999999e-05 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861643+0000", 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1440999999999998e-05 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861797+0000", 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000153688 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861814+0000", 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7269e-05 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.868678+0000", 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068639139999999996 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.868904+0000", 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000225644 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.887308+0000", 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018404594999999999 2024-09-16T04:02:19.618 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:516 3.cs0 3:35c140c5:::smithi18831615-48:head [read 0~2211840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861724+0000", 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.694664590999999, 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030702040999999999, 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.619 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 516 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861724+0000", 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861724+0000", 2024-09-16T04:02:19.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861725+0000", 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6360000000000001e-06 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861735+0000", 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2229999999999996e-06 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861757+0000", 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2069000000000002e-05 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861774+0000", 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7379000000000001e-05 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861880+0000", 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010627 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861888+0000", 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2199999999999992e-06 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.869811+0000", 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007922208 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.870456+0000", 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00064530399999999997 2024-09-16T04:02:19.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.892426+0000", 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021969731999999999 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:540 3.fs0 3:ff708987:::smithi18831615-20:head [copy-from ver 13] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.068918+0000", 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.487470704, 2024-09-16T04:02:19.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034648133999999997, 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 540 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068918+0000", 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068918+0000", 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068920+0000", 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.717e-06 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068934+0000", 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3472000000000001e-05 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068966+0000", 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2085000000000002e-05 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.068981+0000", 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069050+0000", 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8392000000000005e-05 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069121+0000", 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1513000000000005e-05 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.094700+0000", 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025578430999999999 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.101389+0000", 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066896339999999999 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.103448+0000", 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020585209999999998 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.103566+0000", 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117923 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:541 3.fs0 3:ff708987:::smithi18831615-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.069017+0000", 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.487371412999998, 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032194999000000002, 2024-09-16T04:02:19.630 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 541 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069017+0000", 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069017+0000", 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069018+0000", 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0130000000000001e-06 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069023+0000", 2024-09-16T04:02:19.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8369999999999996e-06 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069036+0000", 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2872e-05 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069045+0000", 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9400000000000008e-06 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069236+0000", 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000191101 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.069260+0000", 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3986e-05 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.100647+0000", 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031387198999999998 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.100938+0000", 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000290933 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.101212+0000", 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027411800000000001 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.635 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:649 3.cs0 3:30f8ffd7:::smithi18831615-11:head [copy-from ver 51] snapc 1b=[1b,1a,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.000111+0000", 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.556277590000001, 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.047605886, 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 649 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.636 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000111+0000", 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000111+0000", 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000112+0000", 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8000000000000004e-07 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000118+0000", 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5200000000000003e-06 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000128+0000", 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0302000000000001e-05 2024-09-16T04:02:19.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000136+0000", 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5970000000000003e-06 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.007587+0000", 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074513799999999996 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.007721+0000", 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013325300000000001 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.042660+0000", 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034939323000000001 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.046875+0000", 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042154849999999997 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.046987+0000", 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011165100000000001 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.047717+0000", 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072949500000000003 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:650 3.cs0 3:30f8ffd7:::smithi18831615-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.000146+0000", 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.556242704999999, 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.044688439000000003, 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 650 2024-09-16T04:02:19.642 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000146+0000", 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000146+0000", 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000146+0000", 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000152+0000", 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2630000000000003e-06 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000161+0000", 2024-09-16T04:02:19.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6600000000000007e-06 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.000169+0000", 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8790000000000003e-06 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.007999+0000", 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078302379999999998 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.008035+0000", 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5336000000000002e-05 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.044510+0000", 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.036475664999999997 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.044663+0000", 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000152793 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.044834+0000", 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017088499999999999 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:706 3.cs0 3:30f8ffd7:::smithi18831615-11:head [copy-from ver 71] snapc 1d=[1d,1c,1b,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.000875+0000", 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.555512814, 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.051306851000000001, 2024-09-16T04:02:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 706 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000875+0000", 2024-09-16T04:02:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000875+0000", 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000879+0000", 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7100000000000001e-06 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000901+0000", 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1937999999999999e-05 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000938+0000", 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6600999999999997e-05 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.000956+0000", 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8437e-05 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001117+0000", 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016098499999999999 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001131+0000", 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3597e-05 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.004495+0000", 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033642619999999998 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.004767+0000", 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000271625 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.037080+0000", 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032313596 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.045271+0000", 2024-09-16T04:02:19.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081910279999999995 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.052078+0000", 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068070939999999996 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.052182+0000", 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103978 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.653 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:707 3.cs0 3:30f8ffd7:::smithi18831615-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:01.001018+0000", 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.555370451000002, 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.043917128999999999, 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 707 2024-09-16T04:02:19.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001018+0000", 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001018+0000", 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001019+0000", 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.029e-06 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001028+0000", 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9069999999999996e-06 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001047+0000", 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8890999999999999e-05 2024-09-16T04:02:19.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001058+0000", 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1242e-05 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001228+0000", 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017024499999999999 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.001237+0000", 2024-09-16T04:02:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1160000000000005e-06 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.005191+0000", 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039534160000000004 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.005350+0000", 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000159273 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.044337+0000", 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.038987496000000003 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.044624+0000", 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028676400000000001 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:01.044935+0000", 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031074999999999999 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.659 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:904 3.fs0 3:f22e48d8:::smithi18831615-33:head [stat,delete] snapc 24=[24,23,21,20,1f,1e,1d,1b,19,18,17,16,15,11,f,e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:13.829559+0000", 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.7268293420000003, 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030138977000000001, 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.660 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 904 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829559+0000", 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829559+0000", 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829562+0000", 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2229999999999999e-06 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829575+0000", 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2903e-05 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829616+0000", 2024-09-16T04:02:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0672e-05 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.829637+0000", 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0803000000000001e-05 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831227+0000", 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001589998 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.831490+0000", 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026349099999999997 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.833666+0000", 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002175476 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.850150+0000", 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016484114000000001 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.859508+0000", 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093582509999999997 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:13.859698+0000", 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000190046 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:954 3.cs0 3:30f8ffd7:::smithi18831615-11:head [copy-from ver 152] snapc 28=[28,27,26,25,24,23,21,20,1f,1d,1b,19,18,17,16,15,11,f,e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:18.740797+0000", 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.81559132099999998, 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033855812999999998, 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740797+0000", 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740797+0000", 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740800+0000", 2024-09-16T04:02:19.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7269999999999999e-06 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740813+0000", 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3305999999999999e-05 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740844+0000", 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0781999999999998e-05 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740860+0000", 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5797000000000001e-05 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.741027+0000", 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016765500000000001 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.741328+0000", 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000301234 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.758908+0000", 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017579860999999999 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.766784+0000", 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078754140000000007 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.774426+0000", 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076419180000000001 2024-09-16T04:02:19.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.774653+0000", 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022711899999999999 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:955 3.cs0 3:30f8ffd7:::smithi18831615-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:18.740894+0000", 2024-09-16T04:02:19.671 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.81549474, 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025895017999999999, 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 955 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740894+0000", 2024-09-16T04:02:19.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740894+0000", 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740895+0000", 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.043e-06 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:19.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740899+0000", 2024-09-16T04:02:19.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8799999999999999e-06 2024-09-16T04:02:19.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:19.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740919+0000", 2024-09-16T04:02:19.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9692999999999999e-05 2024-09-16T04:02:19.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:19.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.740932+0000", 2024-09-16T04:02:19.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2830000000000001e-05 2024-09-16T04:02:19.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.741751+0000", 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081918599999999996 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.741834+0000", 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2534000000000006e-05 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.766470+0000", 2024-09-16T04:02:19.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024636464 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.766621+0000", 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000150996 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:18.766789+0000", 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016739200000000001 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:19.678 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:19.679 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:19.680 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:02:19.681 INFO:tasks.ceph.osd.10.smithi117.stderr:2024-09-16T04:02:19.677+0000 7f85946e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26401) UID: 0 2024-09-16T04:02:19.712 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:19.712 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:19.733 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-09-16T04:02:19.757 INFO:tasks.rados.rados.0.smithi188.stdout:380: done (1 left) 2024-09-16T04:02:19.757 INFO:tasks.rados.rados.0.smithi188.stdout:381: delete oid 41 current snap is 39 2024-09-16T04:02:19.757 INFO:tasks.rados.rados.0.smithi188.stdout:379: finishing write tid 1 to smithi18831615-7 2024-09-16T04:02:19.757 INFO:tasks.rados.rados.0.smithi188.stdout:379: finishing write tid 2 to smithi18831615-7 2024-09-16T04:02:19.757 INFO:tasks.rados.rados.0.smithi188.stdout:379: finishing write tid 3 to smithi18831615-7 2024-09-16T04:02:19.757 INFO:tasks.rados.rados.0.smithi188.stdout:379: oid 7 updating version 0 to 113 2024-09-16T04:02:19.757 INFO:tasks.rados.rados.0.smithi188.stdout:379: oid 7 updating version 113 to 114 2024-09-16T04:02:19.758 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 7 v 114 (ObjNum 150 snap 39 seq_num 150) dirty exists 2024-09-16T04:02:19.758 INFO:tasks.rados.rados.0.smithi188.stdout:379: left oid 7 (ObjNum 150 snap 39 seq_num 150) 2024-09-16T04:02:19.766 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:19.766 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:19.776 INFO:tasks.rados.rados.0.smithi188.stdout:379: done (1 left) 2024-09-16T04:02:19.776 INFO:tasks.rados.rados.0.smithi188.stdout:381: done (0 left) 2024-09-16T04:02:19.776 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 36 current snap is 39 2024-09-16T04:02:19.776 INFO:tasks.rados.rados.0.smithi188.stdout:382: seq_num 152 ranges {376832=147456} 2024-09-16T04:02:19.784 INFO:tasks.rados.rados.0.smithi188.stdout:382: writing smithi18831615-36 from 376832 to 524288 tid 1 2024-09-16T04:02:19.784 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 43 current snap is 39 2024-09-16T04:02:19.784 INFO:tasks.rados.rados.0.smithi188.stdout:383: seq_num 153 ranges {2342912=163840} 2024-09-16T04:02:19.784 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:02:19.780+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:02:19.795 INFO:tasks.rados.rados.0.smithi188.stdout:383: writing smithi18831615-43 from 2342912 to 2506752 tid 1 2024-09-16T04:02:19.795 INFO:tasks.rados.rados.0.smithi188.stdout:384: copy_from oid 24 from oid 32 current snap is 39 2024-09-16T04:02:19.795 INFO:tasks.rados.rados.0.smithi188.stdout:382: finishing write tid 1 to smithi18831615-36 2024-09-16T04:02:19.795 INFO:tasks.rados.rados.0.smithi188.stdout:382: finishing write tid 2 to smithi18831615-36 2024-09-16T04:02:19.795 INFO:tasks.rados.rados.0.smithi188.stdout:385: copy_from oid 17 from oid 2 current snap is 39 2024-09-16T04:02:19.795 INFO:tasks.rados.rados.0.smithi188.stdout:386: read oid 8 snap -1 2024-09-16T04:02:19.795 INFO:tasks.rados.rados.0.smithi188.stdout:386: expect (ObjNum 109 snap 25 seq_num 109) 2024-09-16T04:02:19.796 INFO:tasks.rados.rados.0.smithi188.stdout:387: snap_remove snap 15 2024-09-16T04:02:19.798 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:19.817 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:19.818 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:19.833 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:19.833 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:19.833 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-09-16T04:02:19.833 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:19.838 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:19.847 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:02:19.859 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:19.859 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:19.878 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:19.879 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:19.882 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:19.880+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:19.895 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:19.917 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:19.918 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:19.939 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:19.940 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:19.960 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:19.960 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:19.980 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:19.983 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:19.977+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:20.002 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:20.003 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:20.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:02:20.000+0000 7f1dc1cb1640 0 -- 172.21.15.37:0/2373759911 >> [v2:172.21.15.117:6808/1375514506,v1:172.21.15.117:6809/1375514506] conn(0x7f1dbc221bf0 msgr2=0x7f1dbc223fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T04:02:20.006 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:20.006 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-09-16T04:02:20.006 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-09-16T04:02:20.007 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-09-16T04:02:20.007 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:20.021 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:02:20.085 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:20.080+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:108 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 58.210165033000003, 2024-09-16T04:02:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014725059, 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 108 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:02:20.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933256+0000", 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2399999999999998e-07 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933259+0000", 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.937e-06 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933267+0000", 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6229999999999997e-06 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.933273+0000", 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936240+0000", 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002967365 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.936330+0000", 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0200999999999999e-05 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.940982+0000", 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046523210000000001 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.941058+0000", 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5798999999999997e-05 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:21.947981+0000", 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006922615 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.153 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:314 3.5s0 3:a51a87ac:::smithi18831615-36:head [append 0~491520] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 54.132773667999999, 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013929950999999999, 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.154 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 314 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010647+0000", 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.010651+0000", 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.467e-06 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011198+0000", 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054779799999999995 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011237+0000", 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8092000000000001e-05 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011254+0000", 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7232999999999999e-05 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011454+0000", 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020057300000000001 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.011623+0000", 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016875899999999999 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.014078+0000", 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024548510000000001 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024181+0000", 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010102877 2024-09-16T04:02:20.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024366+0000", 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018482199999999999 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024577+0000", 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021147900000000001 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:316 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (56) in=7b] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 54.126007973999997, 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010644074999999999, 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 316 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.160 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017413+0000", 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017415+0000", 2024-09-16T04:02:20.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.99e-06 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017424+0000", 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4129999999999995e-06 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017444+0000", 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0115999999999998e-05 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017463+0000", 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8199999999999999e-05 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022150+0000", 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046871079999999997 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.022414+0000", 2024-09-16T04:02:20.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000264491 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.023228+0000", 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081356500000000003 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.025417+0000", 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021892959999999999 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.027816+0000", 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023989509999999999 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.028057+0000", 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000240945 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:317 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-16T04:02:20.165 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 54.125896900000001, 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017770083999999998, 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 317 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:02:20.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017524+0000", 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017526+0000", 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017533+0000", 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1960000000000002e-06 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017555+0000", 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2157000000000001e-05 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.017574+0000", 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8298999999999999e-05 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024692+0000", 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071183920000000003 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.024964+0000", 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027219700000000001 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.029509+0000", 2024-09-16T04:02:20.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045448850000000002 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.031984+0000", 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024747929999999999 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:26.035294+0000", 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033101850000000002 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.170 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:510 3.5s0 3:a51a87ac:::smithi18831615-36:head [copy-from ver 36] snapc 16=[16,15,14,13,12,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861100+0000", 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.282320455000001, 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.041038472999999999, 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.171 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 510 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861100+0000", 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861100+0000", 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861104+0000", 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3799999999999998e-06 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861121+0000", 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7459e-05 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861172+0000", 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.126e-05 2024-09-16T04:02:20.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861200+0000", 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7138999999999999e-05 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861474+0000", 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027421100000000001 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861498+0000", 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4468999999999999e-05 2024-09-16T04:02:20.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.866976+0000", 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054775709999999997 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867232+0000", 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025570900000000001 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.895930+0000", 2024-09-16T04:02:20.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028698860999999999 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.900460+0000", 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004529178 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.901938+0000", 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014787699999999999 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.902139+0000", 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000200466 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:511 3.5s0 3:a51a87ac:::smithi18831615-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:50.861259+0000", 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.282161630000001, 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.038909821999999997, 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.177 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 511 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861259+0000", 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861259+0000", 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861261+0000", 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6029999999999999e-06 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861269+0000", 2024-09-16T04:02:20.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8420000000000005e-06 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861303+0000", 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4662000000000001e-05 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861322+0000", 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902e-05 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861670+0000", 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034817399999999998 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.861687+0000", 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6626e-05 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867660+0000", 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059733540000000002 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.867771+0000", 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011105 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.899915+0000", 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032144104 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.900038+0000", 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000122679 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:50.900169+0000", 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013082600000000001 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.183 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s3 66/33 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=66'32, trim_to=0'0, roll_forward_to=63'30))", 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:51.737038+0000", 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.406382388000001, 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011633279999999999, 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737038+0000", 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737038+0000", 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737039+0000", 2024-09-16T04:02:20.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737092+0000", 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3233999999999999e-05 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737117+0000", 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4782000000000001e-05 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737140+0000", 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2898e-05 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737324+0000", 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018401999999999999 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.737349+0000", 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5021999999999999e-05 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.748605+0000", 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011255644 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:51.748672+0000", 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7096999999999998e-05 2024-09-16T04:02:20.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:563 3.5s0 3:a37b04c5:::smithi18831615-47:head [append 0~0] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:52.761739+0000", 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.381682081000001, 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010909632000000001, 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.189 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 563 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761739+0000", 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761739+0000", 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761742+0000", 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9579999999999999e-06 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761752+0000", 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0662999999999999e-05 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761775+0000", 2024-09-16T04:02:20.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2293000000000002e-05 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761788+0000", 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3074000000000001e-05 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761939+0000", 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015136500000000001 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.762175+0000", 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023601300000000001 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.763620+0000", 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001444877 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.768564+0000", 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049439870000000004 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.772454+0000", 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003889783 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.772648+0000", 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019461899999999999 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.194 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:564 3.5s0 3:a37b04c5:::smithi18831615-47:head [rollback e] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:52.761815+0000", 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.381605482000001, 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015990183000000002, 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 564 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761815+0000", 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761815+0000", 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761816+0000", 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6499999999999998e-07 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761821+0000", 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761834+0000", 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3643e-05 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761842+0000", 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1130000000000001e-06 2024-09-16T04:02:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.765701+0000", 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038581029999999999 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.765961+0000", 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026021099999999999 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.767423+0000", 2024-09-16T04:02:20.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001461979 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.772787+0000", 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053647039999999997 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.777603+0000", 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048160119999999997 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.777806+0000", 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020206900000000001 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:565 3.5s0 3:a37b04c5:::smithi18831615-47:head [append 0~0] snapc 17=[17,16,15,14,13,11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:52.761896+0000", 2024-09-16T04:02:20.200 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.381524508999998, 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017049812000000001, 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 565 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761896+0000", 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761896+0000", 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761897+0000", 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1399999999999996e-07 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761902+0000", 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4190000000000001e-06 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761915+0000", 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2998e-05 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.761924+0000", 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1750000000000005e-06 2024-09-16T04:02:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.769117+0000", 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071931879999999997 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.769446+0000", 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032917200000000003 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.770752+0000", 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013064000000000001 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.773481+0000", 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027282980000000001 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.778765+0000", 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052845219999999998 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:52.778946+0000", 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018092600000000001 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:618 3.5s0 3:a51a87ac:::smithi18831615-36:head [copy-from ver 31] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928147+0000", 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.215274033, 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015153856, 2024-09-16T04:02:20.206 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 618 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928147+0000", 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928147+0000", 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928151+0000", 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8299999999999998e-06 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928167+0000", 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6152999999999999e-05 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928203+0000", 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6554000000000003e-05 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928226+0000", 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3142999999999998e-05 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928475+0000", 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024883300000000001 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928758+0000", 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000282883 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.938731+0000", 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099730040000000006 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.942778+0000", 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040466529999999999 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.943074+0000", 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029642900000000002 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.943301+0000", 2024-09-16T04:02:20.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000226374 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:619 3.5s0 3:a51a87ac:::smithi18831615-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:55.928283+0000", 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.215137525999999, 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013685522, 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.212 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 619 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928283+0000", 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928283+0000", 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928285+0000", 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.623e-06 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928292+0000", 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2400000000000001e-06 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928316+0000", 2024-09-16T04:02:20.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3462e-05 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.928335+0000", 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9575000000000001e-05 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.929213+0000", 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00087792499999999999 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for blocked object", 2024-09-16T04:02:20.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.929415+0000", 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020199000000000001 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941419+0000", 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012004182 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941669+0000", 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000249924 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:55.941969+0000", 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029960099999999998 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:632 3.5s0 3:a37b04c5:::smithi18831615-47:head [create,setxattr [OXS[Rf]kSmSVZkcqrPoTRUsq[o]epTblNRaQggt (1858) in=40b,setxattr QrooaVmb\\N]q[ho]kgWWcr\\eocaQRtZ_`]Qsipqt (174) in=40b,setxattr Ydefbt^YQ\\dpmjectRYX^qRe[eQdQqbZhllc (770) in=36b,setxattr TV\\gUfWjiSV[PabStYoPOXcXUVsSaXdhhRnnhpfs (454) in=40b,setxattr m_O\\\\icSfcPOnoNRZ]O`X]X`ZaVOXegXNVmPglmp (734) in=40b,setxattr mf`_l`a[Y`agdjhtOpoWmnWT^dS[ScSdZWamrcjp (699) in=40b,setxattr hPm]lVO[\\[iefNWiNnaWNQr (770) in=23b,setxattr gkna_]qPX[S`[_g_ifYo]hUlgScp_jnl[^WpdheP (1614) in=40b,setxattr YSndfaQqlVQQXmZiNjPrZZa`glYedsbkkcroSttS (1333) in=40b,setxattr VVliPgssabe (770) in=11b,setxattr kdelpR\\kNn^\\r\\aeabqa\\jcQSaOXgNlVTroTaodb (1610) in=40b,setxattr a[YZZ[_nQae][Ujatlhnen_QorP_PROiQkaafPl^ (1754) in=40b,setxattr Y\\iYcXa`paqfZggTnSmaVZ[Yt`te\\rdepg^foUUX (12280) in=40b,setxattr WPicPk_V\\Z]`USR]\\Xq (770) in=19b,setxattr gqtnZjUUjhkmnQqhohYVhkliPOppPSXTshf_sVbq (877) in=40b,setxattr Nadgio[acb^P_fdps]bigQrctbcktsZdTe]Z_hhb (8628) in=40b,setxattr oUUbUt^qtWedq]`NoRbZf[jPhdNrRbjTU`m^jbpe (666) in=40b,setxattr UUhYqmbd_S[all[PkPg\\odhP\\gkshjetOj][^OWq (16140) in=40b,setxattr WSTftae_cS`q^sZiZQskrTnpRX^Tkmid^ZbYOeTo (1997) in=40b,setxattr bdcqYYhcTQq^O[TjlUb[oTXa_hoaQStVglem^VqX (981) in=40b] snapc 1a=[1a,19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:56.940054+0000", 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.203366891000002, 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015103711000000001, 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.217 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 632 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940054+0000", 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940054+0000", 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940058+0000", 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9960000000000004e-06 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940158+0000", 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9825000000000001e-05 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940209+0000", 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0788000000000002e-05 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940235+0000", 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6137999999999999e-05 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940406+0000", 2024-09-16T04:02:20.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017153000000000001 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.940423+0000", 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7241999999999999e-05 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.944927+0000", 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045031070000000001 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.945295+0000", 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036849799999999998 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.948503+0000", 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003208397 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.954274+0000", 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057702350000000003 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.954431+0000", 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015692500000000001 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:56.955158+0000", 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072703000000000004 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.223 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:665 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.148626+0000", 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.994794665000001, 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014036061000000001, 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 665 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148626+0000", 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148626+0000", 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148630+0000", 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4139999999999998e-06 2024-09-16T04:02:20.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148647+0000", 2024-09-16T04:02:20.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7419e-05 2024-09-16T04:02:20.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148694+0000", 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7209999999999997e-05 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148717+0000", 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2364000000000001e-05 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148833+0000", 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000116414 2024-09-16T04:02:20.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148974+0000", 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014129100000000001 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.162662+0000", 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013687949 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:666 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.148763+0000", 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.994657716999999, 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0150174, 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 666 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.229 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148763+0000", 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148763+0000", 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148765+0000", 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.466e-06 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148773+0000", 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3620000000000006e-06 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148792+0000", 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8799000000000001e-05 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148807+0000", 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5325e-05 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.156052+0000", 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072448499999999997 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.156280+0000", 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000228536 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.163780+0000", 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007500062 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:667 3.5s0 3:a37b04c5:::smithi18831615-47:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:01:58.148837+0000", 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.994584044, 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015477739000000001, 2024-09-16T04:02:20.233 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 667 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148837+0000", 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148837+0000", 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148838+0000", 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.012e-06 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148847+0000", 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7309999999999999e-06 2024-09-16T04:02:20.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148864+0000", 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7748e-05 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.148878+0000", 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3848e-05 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.157464+0000", 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085862760000000003 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.157707+0000", 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000242215 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:01:58.164315+0000", 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066079090000000004 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.237 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.3s3 94/33 ECSubWrite(tid=319, reqid=osd.13.0:319, at_version=94'76, trim_to=0'0, roll_forward_to=92'74))", 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:19.757855+0000", 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.38556536899999999, 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012832059, 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.757855+0000", 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.757855+0000", 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.757856+0000", 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8699999999999995e-07 2024-09-16T04:02:20.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.757910+0000", 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3542000000000003e-05 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.757931+0000", 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1022999999999998e-05 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.757945+0000", 2024-09-16T04:02:20.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4440000000000001e-05 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.758092+0000", 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000147248 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.758118+0000", 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.584e-05 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.770582+0000", 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012463795 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.770688+0000", 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105584 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.242 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.6s3 94/33 ECSubWrite(tid=24, reqid=osd.0.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=94'15))", 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:19.759354+0000", 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.38406693200000003, 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012206730000000001, 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.759354+0000", 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.759354+0000", 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.759355+0000", 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.759379+0000", 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4224999999999999e-05 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.759397+0000", 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8185999999999998e-05 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.759410+0000", 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3064000000000001e-05 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.759540+0000", 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000129629 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.760363+0000", 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00082304299999999997 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.771427+0000", 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011063765999999999 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.771561+0000", 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013378899999999999 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:970 3.5s0 3:a51a87ac:::smithi18831615-36:head [setxattr _header (57) in=7b] snapc 28=[28,27,26,25,24,21,20,1f,1d,1b,19,18,17,16,15,11,f,e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:19.782634+0000", 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.36078718100000001, 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010480654000000001, 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 970 2024-09-16T04:02:20.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782634+0000", 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782634+0000", 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782634+0000", 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2399999999999997e-07 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782639+0000", 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5499999999999996e-06 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782651+0000", 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1644000000000001e-05 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782660+0000", 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7280000000000007e-06 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.785947+0000", 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032865899999999998 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.786249+0000", 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000302505 2024-09-16T04:02:20.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.787745+0000", 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00149507 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.790401+0000", 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026560339999999998 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.792996+0000", 2024-09-16T04:02:20.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002594973 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.793114+0000", 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118736 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4646.0:971 3.5s0 3:a51a87ac:::smithi18831615-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-09-16T04:02:19.782673+0000", 2024-09-16T04:02:20.252 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.36074771700000002, 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012435224999999999, 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4646", 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 971 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782673+0000", 2024-09-16T04:02:20.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782673+0000", 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782674+0000", 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3200000000000004e-07 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-09-16T04:02:20.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782678+0000", 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7009999999999998e-06 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782689+0000", 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0878e-05 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.782696+0000", 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5870000000000004e-06 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.788417+0000", 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057206599999999998 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.788670+0000", 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025316599999999998 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.793508+0000", 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048377749999999999 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.793649+0000", 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000141485 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-09-16T04:02:19.795108+0000", 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001459141 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-09-16T04:02:20.257 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-09-16T04:02:20.259 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:20.259 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:20.260 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:02:20.260 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:20.256+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:20.266 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:20.266 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:20.360 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:20.356+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:20.399 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:20.400 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:20.414 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:02:20.460 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:20.456+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:20.561 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:20.556+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:20.585 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:20.585 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:20.599 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:02:20.661 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:20.656+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:20.726 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:20.726 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:20.740 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:02:20.750 INFO:tasks.rados.rados.0.smithi188.stdout:387: done (5 left) 2024-09-16T04:02:20.750 INFO:tasks.rados.rados.0.smithi188.stdout:388: snap_remove snap 36 2024-09-16T04:02:20.765 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:20.760+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:20.864 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:20.860+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:20.910 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:20.910 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:20.925 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:02:20.970 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:20.968+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:21.053 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:21.054 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:21.066 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:21.060+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:21.068 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:02:21.166 INFO:tasks.ceph.osd.13.smithi039.stderr:2024-09-16T04:02:21.164+0000 7f65563a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26266) UID: 0 2024-09-16T04:02:21.238 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:21.238 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:21.253 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:02:21.267 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:21.264+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:21.369 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:21.364+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:21.379 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:21.379 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:21.393 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:02:21.469 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-09-16T04:02:21.468+0000 7fd82e930640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26374) UID: 0 2024-09-16T04:02:21.516 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:21.517 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:21.528 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:02:21.570 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:21.568+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:21.671 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:21.668+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:21.698 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:21.698 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:21.712 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:388: done (5 left) 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:389: read oid 41 snap 12 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:389: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:390: rmattr oid 10 current snap is 39 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:390: done (6 left) 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:391: read oid 7 snap -1 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:391: expect (ObjNum 150 snap 39 seq_num 150) 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 22 current snap is 39 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:382: finishing write tid 3 to smithi18831615-36 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:382: oid 36 updating version 0 to 60 2024-09-16T04:02:21.738 INFO:tasks.rados.rados.0.smithi188.stdout:382: oid 36 version 60 is already newer than 59 2024-09-16T04:02:21.739 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 36 v 60 (ObjNum 152 snap 39 seq_num 152) dirty exists 2024-09-16T04:02:21.739 INFO:tasks.rados.rados.0.smithi188.stdout:382: left oid 36 (ObjNum 152 snap 39 seq_num 152) 2024-09-16T04:02:21.739 INFO:tasks.rados.rados.0.smithi188.stdout:392: seq_num 154 ranges {1753088=180224} 2024-09-16T04:02:21.767 INFO:tasks.rados.rados.0.smithi188.stdout:392: writing smithi18831615-22 from 1753088 to 1933312 tid 1 2024-09-16T04:02:21.767 INFO:tasks.rados.rados.0.smithi188.stdout:382: done (7 left) 2024-09-16T04:02:21.767 INFO:tasks.rados.rados.0.smithi188.stdout:393: read oid 18 snap -1 2024-09-16T04:02:21.767 INFO:tasks.rados.rados.0.smithi188.stdout:393: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:02:21.767 INFO:tasks.rados.rados.0.smithi188.stdout:394: delete oid 37 current snap is 39 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:385: finishing copy_from to smithi18831615-17 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:385: got expected ENOENT (src dne) 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:385: finishing copy_from racing read to smithi18831615-17 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:383: finishing write tid 1 to smithi18831615-43 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:383: finishing write tid 2 to smithi18831615-43 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:383: finishing write tid 3 to smithi18831615-43 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:383: oid 43 updating version 0 to 84 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:383: oid 43 version 84 is already newer than 83 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 43 v 84 (ObjNum 153 snap 39 seq_num 153) dirty exists 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:383: left oid 43 (ObjNum 153 snap 39 seq_num 153) 2024-09-16T04:02:21.768 INFO:tasks.rados.rados.0.smithi188.stdout:386: expect (ObjNum 109 snap 25 seq_num 109) 2024-09-16T04:02:21.777 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:02:21.778 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:21.771+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:21.857 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:21.857 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:21.872 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:02:21.873 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:02:21.868+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:02:21.942 INFO:tasks.rados.rados.0.smithi188.stdout:384: finishing copy_from racing read to smithi18831615-24 2024-09-16T04:02:21.942 INFO:tasks.rados.rados.0.smithi188.stdout:384: finishing copy_from to smithi18831615-24 2024-09-16T04:02:21.942 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 24 v 47 (ObjNum 138 snap 34 seq_num 138) dirty exists 2024-09-16T04:02:21.942 INFO:tasks.rados.rados.0.smithi188.stdout:389: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:02:21.959 INFO:tasks.rados.rados.0.smithi188.stdout:391: expect (ObjNum 150 snap 39 seq_num 150) 2024-09-16T04:02:21.974 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:21.972+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:21.993 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:21.993 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:22.007 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:02:22.061 INFO:tasks.rados.rados.0.smithi188.stdout:392: finishing write tid 1 to smithi18831615-22 2024-09-16T04:02:22.062 INFO:tasks.rados.rados.0.smithi188.stdout:392: finishing write tid 2 to smithi18831615-22 2024-09-16T04:02:22.062 INFO:tasks.rados.rados.0.smithi188.stdout:393: expect (ObjNum 76 snap 3 seq_num 76) 2024-09-16T04:02:22.075 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:22.071+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:22.128 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:22.129 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:392: finishing write tid 3 to smithi18831615-22 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:392: oid 22 updating version 0 to 51 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:392: oid 22 updating version 51 to 52 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 22 v 52 (ObjNum 154 snap 39 seq_num 154) dirty exists 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:392: left oid 22 (ObjNum 154 snap 39 seq_num 154) 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:383: done (8 left) 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:384: done (7 left) 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:385: done (6 left) 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:386: done (5 left) 2024-09-16T04:02:22.141 INFO:tasks.rados.rados.0.smithi188.stdout:389: done (4 left) 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:391: done (3 left) 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:392: done (2 left) 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:393: done (1 left) 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:394: done (0 left) 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:395: read oid 47 snap -1 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:395: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:396: read oid 10 snap 12 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:396: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:append oid 25 current snap is 39 2024-09-16T04:02:22.142 INFO:tasks.rados.rados.0.smithi188.stdout:397: seq_num 156 ranges {0=229376} 2024-09-16T04:02:22.143 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:02:22.144 INFO:tasks.rados.rados.0.smithi188.stdout:397: writing smithi18831615-25 from 0 to 229376 tid 1 2024-09-16T04:02:22.144 INFO:tasks.rados.rados.0.smithi188.stdout:398: snap_create 2024-09-16T04:02:22.146 INFO:tasks.rados.rados.0.smithi188.stdout:396: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:02:22.156 INFO:tasks.rados.rados.0.smithi188.stdout:395: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:02:22.175 INFO:tasks.rados.rados.0.smithi188.stdout:397: finishing write tid 1 to smithi18831615-25 2024-09-16T04:02:22.175 INFO:tasks.rados.rados.0.smithi188.stdout:397: finishing write tid 2 to smithi18831615-25 2024-09-16T04:02:22.175 INFO:tasks.rados.rados.0.smithi188.stdout:397: finishing write tid 3 to smithi18831615-25 2024-09-16T04:02:22.175 INFO:tasks.rados.rados.0.smithi188.stdout:397: oid 25 updating version 0 to 31 2024-09-16T04:02:22.175 INFO:tasks.rados.rados.0.smithi188.stdout:397: oid 25 updating version 31 to 32 2024-09-16T04:02:22.175 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 25 v 32 (ObjNum 156 snap 39 seq_num 156) dirty exists 2024-09-16T04:02:22.175 INFO:tasks.rados.rados.0.smithi188.stdout:397: left oid 25 (ObjNum 156 snap 39 seq_num 156) 2024-09-16T04:02:22.176 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:22.172+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:22.277 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:22.272+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:22.280 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:22.280 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:22.290 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:02:22.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:02:22.378 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:22.373+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:22.446 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:22.467 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:22.467 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:22.468 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:22.468 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:22.468 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:02:22.468 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:02:22.468 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:22.479 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-09-16T04:02:22.476+0000 7f2b6be40640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26643) UID: 0 2024-09-16T04:02:22.483 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:02:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:22.488 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:22.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:22.547 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:22.568 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:22.579 INFO:tasks.ceph.osd.3.smithi188.stderr:2024-09-16T04:02:22.576+0000 7fb92b9fb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26640) UID: 0 2024-09-16T04:02:22.591 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:22.592 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:22.611 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:22.611 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:02:22.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:02:22.611 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:02:22.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.611 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/33 ECSubWrite(tid=141, reqid=osd.4.0:141, at_version=66'69, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:02:22.611 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.730939+0000", 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.876010807, 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010155496, 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730939+0000", 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730939+0000", 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730940+0000", 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1960000000000001e-06 2024-09-16T04:02:22.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.730998+0000", 2024-09-16T04:02:22.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8137000000000003e-05 2024-09-16T04:02:22.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731025+0000", 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6749000000000001e-05 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.731044+0000", 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9306000000000001e-05 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733535+0000", 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0024914049999999999 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733563+0000", 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7378000000000001e-05 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740991+0000", 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0074285530000000001 2024-09-16T04:02:22.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.741094+0000", 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000102772 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.617 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.es2 82/33 ECSubWrite(tid=33, reqid=osd.2.0:33, at_version=82'16, trim_to=0'0, roll_forward_to=81'14))", 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:07.778311+0000", 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.82863867, 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013286011, 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778311+0000", 2024-09-16T04:02:22.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778311+0000", 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778311+0000", 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4300000000000003e-07 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778365+0000", 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3604999999999999e-05 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778396+0000", 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1439000000000002e-05 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778418+0000", 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1903e-05 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.782402+0000", 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039834299999999996 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.782428+0000", 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6438999999999999e-05 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.791556+0000", 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0091276570000000008 2024-09-16T04:02:22.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.791597+0000", 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0995000000000003e-05 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 82/33 ECSubWrite(tid=164, reqid=osd.14.0:164, at_version=82'106, trim_to=0'0, roll_forward_to=81'104))", 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:07.778683+0000", 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.828266764, 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012627427, 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778683+0000", 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778683+0000", 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778684+0000", 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7230000000000001e-06 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778735+0000", 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0714999999999998e-05 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778767+0000", 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2106000000000001e-05 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778791+0000", 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4329e-05 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.779820+0000", 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001028805 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.779846+0000", 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5609000000000002e-05 2024-09-16T04:02:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.791203+0000", 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011357423 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.791310+0000", 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106717 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 82/33 ECSubWrite(tid=165, reqid=osd.14.0:165, at_version=82'27, trim_to=0'0, roll_forward_to=81'25))", 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:07.778855+0000", 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.828094772, 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010980382, 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778855+0000", 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778855+0000", 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778856+0000", 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5999999999999991e-07 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778905+0000", 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9827999999999998e-05 2024-09-16T04:02:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778929+0000", 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3356000000000001e-05 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778946+0000", 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7465999999999998e-05 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.779712+0000", 2024-09-16T04:02:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00076617600000000003 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.779739+0000", 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6376e-05 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.789799+0000", 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010060008 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.789835+0000", 2024-09-16T04:02:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6211999999999998e-05 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 88/33 ECSubWrite(tid=274, reqid=osd.4.0:274, at_version=88'147, trim_to=0'0, roll_forward_to=86'143))", 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.831375+0000", 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7755741880000002, 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016755341999999999, 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831375+0000", 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831375+0000", 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831377+0000", 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6029999999999999e-06 2024-09-16T04:02:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831428+0000", 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1115000000000001e-05 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831454+0000", 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6570000000000001e-05 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831476+0000", 2024-09-16T04:02:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1195e-05 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831707+0000", 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023109899999999999 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831732+0000", 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5128999999999999e-05 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848039+0000", 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016306863000000001 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848131+0000", 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1768000000000006e-05 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.2s5 88/33 ECSubWrite(tid=301, reqid=osd.13.0:301, at_version=88'48, trim_to=0'0, roll_forward_to=86'46))", 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.831432+0000", 2024-09-16T04:02:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7755173440000007, 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012619648000000001, 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831432+0000", 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831432+0000", 2024-09-16T04:02:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831433+0000", 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0400000000000005e-07 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831490+0000", 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7058999999999997e-05 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831522+0000", 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2342000000000003e-05 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831558+0000", 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5408999999999999e-05 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831722+0000", 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016417299999999999 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.831744+0000", 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2133e-05 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.843947+0000", 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01220332 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.844052+0000", 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010440799999999999 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.640 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 88/33 ECSubWrite(tid=206, reqid=osd.14.0:206, at_version=88'134, trim_to=0'0, roll_forward_to=86'132))", 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.833594+0000", 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7733557609999995, 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018393553999999999, 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833594+0000", 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833594+0000", 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833594+0000", 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2999999999999999e-07 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833656+0000", 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1401000000000004e-05 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833690+0000", 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4035999999999997e-05 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833713+0000", 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2770000000000001e-05 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.834104+0000", 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039090700000000002 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.834130+0000", 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6325999999999998e-05 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.851876+0000", 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017746485999999999 2024-09-16T04:02:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.851987+0000", 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011079800000000001 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s3 88/33 ECSubWrite(tid=207, reqid=osd.14.0:207, at_version=88'136, trim_to=0'0, roll_forward_to=86'132))", 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.836354+0000", 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7705954199999994, 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026831812999999999, 2024-09-16T04:02:22.645 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836354+0000", 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836354+0000", 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836356+0000", 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.823e-06 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836407+0000", 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1255999999999999e-05 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836446+0000", 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9215000000000003e-05 2024-09-16T04:02:22.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836470+0000", 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3657000000000001e-05 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840285+0000", 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0038152149999999998 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840314+0000", 2024-09-16T04:02:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8666e-05 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863089+0000", 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022775103000000001 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863186+0000", 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6878000000000003e-05 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 88/33 ECSubWrite(tid=276, reqid=client.4646.0:903, at_version=88'149, trim_to=0'0, roll_forward_to=86'143))", 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.837278+0000", 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7696710150000001, 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025639863999999998, 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837278+0000", 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837278+0000", 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837280+0000", 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0099999999999998e-06 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837710+0000", 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00042916299999999998 2024-09-16T04:02:22.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837744+0000", 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3995999999999998e-05 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837782+0000", 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8596000000000001e-05 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837939+0000", 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000157226 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837970+0000", 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0648999999999997e-05 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.862803+0000", 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024832797 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.862918+0000", 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000115427 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.4s3 88/33 ECSubWrite(tid=105, reqid=osd.8.0:104, at_version=0'0, trim_to=0'0, roll_forward_to=88'70))", 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.837508+0000", 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7694418009999993, 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010572163000000001, 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.654 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837508+0000", 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837508+0000", 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837509+0000", 2024-09-16T04:02:22.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9069999999999999e-06 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837537+0000", 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7169e-05 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837568+0000", 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1090000000000002e-05 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.837591+0000", 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3244999999999999e-05 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.838402+0000", 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00081095799999999997 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.838439+0000", 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6572000000000002e-05 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.847974+0000", 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0095354240000000007 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848080+0000", 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010579800000000001 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 88/33 ECSubWrite(tid=208, reqid=osd.14.0:208, at_version=88'37, trim_to=0'0, roll_forward_to=86'35))", 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.840266+0000", 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7666828999999993, 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020653556999999999, 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840266+0000", 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840266+0000", 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840268+0000", 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7770000000000001e-06 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841122+0000", 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085360299999999998 2024-09-16T04:02:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841161+0000", 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9388999999999998e-05 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841184+0000", 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2855999999999999e-05 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841401+0000", 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000216472 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841431+0000", 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0605999999999998e-05 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.860854+0000", 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019423182000000001 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.860920+0000", 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5672000000000004e-05 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.8s3 88/33 ECSubWrite(tid=34, reqid=osd.5.0:34, at_version=88'29, trim_to=0'0, roll_forward_to=84'27))", 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.842066+0000", 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7648837559999997, 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01000616, 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.663 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842066+0000", 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842066+0000", 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842066+0000", 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842162+0000", 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5901000000000002e-05 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842191+0000", 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8541999999999999e-05 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842212+0000", 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0959e-05 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842364+0000", 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015228400000000001 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842389+0000", 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5401999999999998e-05 2024-09-16T04:02:22.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852006+0000", 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.009616332 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852072+0000", 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6015999999999999e-05 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.9s1 88/33 ECSubWrite(tid=209, reqid=osd.14.0:209, at_version=88'39, trim_to=0'0, roll_forward_to=86'35))", 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.844825+0000", 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7621239440000007, 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018431065, 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.844825+0000", 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.844825+0000", 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.844828+0000", 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1059999999999998e-06 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.845353+0000", 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052569800000000001 2024-09-16T04:02:22.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.845400+0000", 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6678999999999998e-05 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.845424+0000", 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4258999999999999e-05 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848452+0000", 2024-09-16T04:02:22.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003027779 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848482+0000", 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0065000000000001e-05 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863208+0000", 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014725668000000001 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863256+0000", 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8810999999999997e-05 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.as2 88/33 ECSubWrite(tid=45, reqid=osd.3.0:45, at_version=88'27, trim_to=0'0, roll_forward_to=86'25))", 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.848128+0000", 2024-09-16T04:02:22.672 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7588210120000003, 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015206130999999999, 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848128+0000", 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848128+0000", 2024-09-16T04:02:22.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848129+0000", 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1900000000000001e-07 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848186+0000", 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6746000000000002e-05 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848219+0000", 2024-09-16T04:02:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2696e-05 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848242+0000", 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2872999999999999e-05 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848455+0000", 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021390800000000001 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.848485+0000", 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9606000000000001e-05 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863259+0000", 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01477345 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863334+0000", 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5933000000000006e-05 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.676 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 88/33 ECSubWrite(tid=277, reqid=osd.4.0:277, at_version=88'151, trim_to=0'0, roll_forward_to=88'147))", 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.849308+0000", 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7576413869999996, 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014032476, 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849308+0000", 2024-09-16T04:02:22.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849308+0000", 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849310+0000", 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1129999999999999e-06 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849396+0000", 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6308000000000001e-05 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849443+0000", 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7008999999999997e-05 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849505+0000", 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1505000000000002e-05 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849718+0000", 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000213382 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849734+0000", 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5255999999999999e-05 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863264+0000", 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013529991999999999 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863340+0000", 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6910999999999996e-05 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 88/33 ECSubWrite(tid=278, reqid=osd.4.0:278, at_version=88'153, trim_to=0'0, roll_forward_to=88'147))", 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.850464+0000", 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7564849579999997, 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014184937, 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.681 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850464+0000", 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850464+0000", 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850466+0000", 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.767e-06 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850530+0000", 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4058000000000002e-05 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850560+0000", 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.021e-05 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850576+0000", 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5169e-05 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852130+0000", 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015544759999999999 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852155+0000", 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5296999999999999e-05 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.864553+0000", 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012397702 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.864649+0000", 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6257999999999999e-05 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.2s5 88/33 ECSubWrite(tid=303, reqid=osd.13.0:301, at_version=0'0, trim_to=0'0, roll_forward_to=88'48))", 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.852492+0000", 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7544575590000004, 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010902214, 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852492+0000", 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852492+0000", 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852494+0000", 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852532+0000", 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8041999999999999e-05 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852566+0000", 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4162000000000003e-05 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852613+0000", 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7236e-05 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852838+0000", 2024-09-16T04:02:22.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022525799999999999 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.852864+0000", 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5443e-05 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863342+0000", 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010478527 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863394+0000", 2024-09-16T04:02:22.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1786000000000002e-05 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.8s3 88/33 ECSubWrite(tid=35, reqid=osd.5.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=88'29))", 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.853831+0000", 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7531182209999994, 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0096835859999999992, 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.853831+0000", 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.853831+0000", 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.853833+0000", 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.88e-06 2024-09-16T04:02:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.853862+0000", 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9201999999999999e-05 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.853896+0000", 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3713000000000001e-05 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.853963+0000", 2024-09-16T04:02:22.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7391999999999994e-05 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.854153+0000", 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018940100000000001 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.854187+0000", 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4010000000000001e-05 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863448+0000", 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0092614220000000001 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.863515+0000", 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6565999999999999e-05 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 92/33 ECSubWrite(tid=296, reqid=osd.4.0:296, at_version=92'161, trim_to=0'0, roll_forward_to=91'157))", 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.737882+0000", 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.8690669629999999, 2024-09-16T04:02:22.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014110846, 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737882+0000", 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737882+0000", 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737883+0000", 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0270000000000001e-06 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737939+0000", 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5928999999999999e-05 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737963+0000", 2024-09-16T04:02:22.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3938999999999999e-05 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737985+0000", 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.139e-05 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.739989+0000", 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0020045839999999998 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.740017+0000", 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7492999999999999e-05 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.751883+0000", 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011866214 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.751993+0000", 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011027 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.698 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs2 92/33 ECSubWrite(tid=300, reqid=osd.4.0:300, at_version=92'165, trim_to=0'0, roll_forward_to=92'161))", 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.755886+0000", 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.8510630849999998, 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012608365999999999, 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755886+0000", 2024-09-16T04:02:22.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755886+0000", 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755888+0000", 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.437e-06 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:22.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755932+0000", 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4311999999999999e-05 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755962+0000", 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9658999999999998e-05 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755982+0000", 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0123e-05 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:22.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.760454+0000", 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044722950000000003 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.760480+0000", 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6046000000000001e-05 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.768335+0000", 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.007854527 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:22.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:22.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.768495+0000", 2024-09-16T04:02:22.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015996700000000001 2024-09-16T04:02:22.703 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.703 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.703 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.703 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:22.703 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:22.703 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:22.705 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:22.705 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:22.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:02:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:22.706 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:22.707 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:22.707 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:22.708 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-16T04:02:22.701+0000 7f6a72611640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26396) UID: 0 2024-09-16T04:02:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:22.723 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:22.748 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:22.771 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:22.774 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:02:22.774 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:02:22.774 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-16T04:02:22.774 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-16T04:02:22.778 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-09-16T04:02:22.787 INFO:tasks.rados.rados.0.smithi188.stdout:395: done (3 left) 2024-09-16T04:02:22.787 INFO:tasks.rados.rados.0.smithi188.stdout:396: done (2 left) 2024-09-16T04:02:22.787 INFO:tasks.rados.rados.0.smithi188.stdout:397: done (1 left) 2024-09-16T04:02:22.787 INFO:tasks.rados.rados.0.smithi188.stdout:398: done (0 left) 2024-09-16T04:02:22.787 INFO:tasks.rados.rados.0.smithi188.stdout:399: copy_from oid 18 from oid 12 current snap is 40 2024-09-16T04:02:22.787 INFO:tasks.rados.rados.0.smithi188.stdout: waiting on 1 2024-09-16T04:02:22.810 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T04:02:22.803+0000 7f5c205c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26371) UID: 0 2024-09-16T04:02:22.811 INFO:tasks.rados.rados.0.smithi188.stdout:399: finishing copy_from racing read to smithi18831615-18 2024-09-16T04:02:22.817 INFO:tasks.rados.rados.0.smithi188.stdout:399: finishing copy_from to smithi18831615-18 2024-09-16T04:02:22.817 INFO:tasks.rados.rados.0.smithi188.stdout:update_object_version oid 18 v 149 (ObjNum 101 snap 21 seq_num 101) dirty exists 2024-09-16T04:02:22.817 INFO:tasks.rados.rados.0.smithi188.stdout:399: done (0 left) 2024-09-16T04:02:22.828 INFO:tasks.rados.rados.0.smithi188.stderr:0 errors. 2024-09-16T04:02:22.828 INFO:tasks.rados.rados.0.smithi188.stderr: 2024-09-16T04:02:22.837 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:02:22.845 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:22.845 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:22.845 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:02:22.845 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:22.861 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:02:22.907 INFO:tasks.ceph.osd.11.smithi188.stderr:2024-09-16T04:02:22.904+0000 7f6cfeb23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26645) UID: 0 2024-09-16T04:02:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:22.952 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:22.953 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-16T04:02:22.991 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:22.991 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:22.991 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:02:22.991 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:02:22.991 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:23.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:02:23.007 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:23.005+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:23.107 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:23.107 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:23.107 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:23.107 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:23.107 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:23.107 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:23.108 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:23.108 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:23.108 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:23.108 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:23.108 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:23.108 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:23.109 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:23.105+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:23.111 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:23.111 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-16T04:02:23.134 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:23.134 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4371.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 76.413300832000004, 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01704027, 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4371", 2024-09-16T04:02:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.37:0/2503621667", 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716687+0000", 2024-09-16T04:02:23.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716696+0000", 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716711+0000", 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4676e-05 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716743+0000", 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1835000000000003e-05 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716769+0000", 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6327999999999999e-05 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.716973+0000", 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020410900000000001 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:02:23.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.717006+0000", 2024-09-16T04:02:23.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3084999999999999e-05 2024-09-16T04:02:23.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.724942+0000", 2024-09-16T04:02:23.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0079355829999999995 2024-09-16T04:02:23.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:02:23.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.724964+0000", 2024-09-16T04:02:23.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1804999999999999e-05 2024-09-16T04:02:23.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730286+0000", 2024-09-16T04:02:23.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053221689999999999 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730641+0000", 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035484999999999998 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.730957+0000", 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0003165 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.732250+0000", 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001293039 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733535+0000", 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001284477 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733616+0000", 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0964999999999995e-05 2024-09-16T04:02:23.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:06.733727+0000", 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011134 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.146 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:458 3.8s0 3:1a33c5ec:::smithi18831615-27:head [append 0~753664] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:23.147 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.776616+0000", 2024-09-16T04:02:23.147 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.353371635999999, 2024-09-16T04:02:23.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022631235, 2024-09-16T04:02:23.147 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.147 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.147 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:23.147 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:23.148 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:23.148 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:23.148 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 458 2024-09-16T04:02:23.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.149 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.776616+0000", 2024-09-16T04:02:23.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.776616+0000", 2024-09-16T04:02:23.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.776619+0000", 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.456e-06 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.777851+0000", 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001231418 2024-09-16T04:02:23.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.777889+0000", 2024-09-16T04:02:23.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8513999999999998e-05 2024-09-16T04:02:23.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.777909+0000", 2024-09-16T04:02:23.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9806e-05 2024-09-16T04:02:23.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.778069+0000", 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016032500000000001 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.778084+0000", 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4606e-05 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782203+0000", 2024-09-16T04:02:23.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041192720000000002 2024-09-16T04:02:23.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:23.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782403+0000", 2024-09-16T04:02:23.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000199643 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.785240+0000", 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0028370359999999998 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.798715+0000", 2024-09-16T04:02:23.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013474709 2024-09-16T04:02:23.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:23.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.798949+0000", 2024-09-16T04:02:23.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023443800000000001 2024-09-16T04:02:23.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.799247+0000", 2024-09-16T04:02:23.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029801200000000001 2024-09-16T04:02:23.160 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.161 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.161 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.162 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:459 3.8s0 3:1a33c5ec:::smithi18831615-27:head [append 0~720896] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:23.162 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.780773+0000", 2024-09-16T04:02:23.162 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.349214189999998, 2024-09-16T04:02:23.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020809588, 2024-09-16T04:02:23.163 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.163 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.163 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:23.163 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:23.164 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:23.164 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:23.164 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 459 2024-09-16T04:02:23.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.164 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.780773+0000", 2024-09-16T04:02:23.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.780773+0000", 2024-09-16T04:02:23.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.780776+0000", 2024-09-16T04:02:23.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3530000000000002e-06 2024-09-16T04:02:23.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782262+0000", 2024-09-16T04:02:23.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0014862779999999999 2024-09-16T04:02:23.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782301+0000", 2024-09-16T04:02:23.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8977999999999998e-05 2024-09-16T04:02:23.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782323+0000", 2024-09-16T04:02:23.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2311999999999999e-05 2024-09-16T04:02:23.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.788336+0000", 2024-09-16T04:02:23.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060125550000000002 2024-09-16T04:02:23.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:23.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.788625+0000", 2024-09-16T04:02:23.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00028906500000000003 2024-09-16T04:02:23.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.791136+0000", 2024-09-16T04:02:23.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002511503 2024-09-16T04:02:23.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.174 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.174 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.799394+0000", 2024-09-16T04:02:23.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082580050000000006 2024-09-16T04:02:23.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:23.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.801480+0000", 2024-09-16T04:02:23.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002086077 2024-09-16T04:02:23.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.801583+0000", 2024-09-16T04:02:23.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000102462 2024-09-16T04:02:23.176 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.176 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.177 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.177 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:460 3.8s0 3:1a33c5ec:::smithi18831615-27:head [append 0~49152] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:23.177 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.782369+0000", 2024-09-16T04:02:23.177 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.347618756999999, 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019757647, 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 460 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782369+0000", 2024-09-16T04:02:23.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782369+0000", 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782371+0000", 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7829999999999999e-06 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782485+0000", 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000114314 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782510+0000", 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5445000000000001e-05 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782532+0000", 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1869e-05 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.794174+0000", 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011642199000000001 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.794422+0000", 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024799400000000002 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.795963+0000", 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015403019999999999 2024-09-16T04:02:23.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.799397+0000", 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034340759999999999 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.801980+0000", 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025832889999999999 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.802126+0000", 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014637600000000001 2024-09-16T04:02:23.182 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:461 3.8s0 3:1a33c5ec:::smithi18831615-27:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,e,d,c,a,7] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.782561+0000", 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.347426640999998, 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020044916999999999, 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:23.183 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 461 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782561+0000", 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782561+0000", 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782562+0000", 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4899999999999999e-06 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782585+0000", 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2918999999999999e-05 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782605+0000", 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9502999999999999e-05 2024-09-16T04:02:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782619+0000", 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4459e-05 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.796957+0000", 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014337242 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.797155+0000", 2024-09-16T04:02:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019887999999999999 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.798133+0000", 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0009780870000000001 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.799665+0000", 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001531632 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.802456+0000", 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027904620000000001 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.802606+0000", 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015024299999999999 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:462 3.8s0 3:1a33c5ec:::smithi18831615-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:45.782640+0000", 2024-09-16T04:02:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.347347605000003, 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028545022, 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 462 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782640+0000", 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782640+0000", 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782641+0000", 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1570000000000001e-06 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782647+0000", 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3430000000000001e-06 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782668+0000", 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0292000000000002e-05 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.782683+0000", 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5362999999999998e-05 2024-09-16T04:02:23.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.800227+0000", 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017543616000000001 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.800355+0000", 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012834000000000001 2024-09-16T04:02:23.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.803488+0000", 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003132485 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.803618+0000", 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130514 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:45.811185+0000", 2024-09-16T04:02:23.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0075669120000000003 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 66/33 ECSubWrite(tid=100, reqid=osd.14.0:100, at_version=66'64, trim_to=0'0, roll_forward_to=65'62))", 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.723674+0000", 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.406314010999999, 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016421259000000001, 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723674+0000", 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723674+0000", 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723674+0000", 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9699999999999996e-07 2024-09-16T04:02:23.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723731+0000", 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723761+0000", 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9325000000000001e-05 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723783+0000", 2024-09-16T04:02:23.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2429000000000002e-05 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.723985+0000", 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020243099999999999 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724019+0000", 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3129000000000001e-05 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.738471+0000", 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014452385 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740095+0000", 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001623869 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 66/33 ECSubWrite(tid=101, reqid=osd.14.0:101, at_version=66'66, trim_to=0'0, roll_forward_to=65'62))", 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.724901+0000", 2024-09-16T04:02:23.198 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.405086858000001, 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015289522, 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724901+0000", 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724901+0000", 2024-09-16T04:02:23.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724903+0000", 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9099999999999999e-06 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724955+0000", 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2420999999999998e-05 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.724988+0000", 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3039e-05 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.725012+0000", 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3794999999999998e-05 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.726248+0000", 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001236516 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.726276+0000", 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8132000000000001e-05 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740105+0000", 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013828653 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740190+0000", 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5055999999999993e-05 2024-09-16T04:02:23.202 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s3 66/33 ECSubWrite(tid=139, reqid=osd.4.0:139, at_version=66'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.727731+0000", 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.402256651999998, 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012696292, 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727731+0000", 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727731+0000", 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727731+0000", 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8599999999999998e-07 2024-09-16T04:02:23.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727790+0000", 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8995e-05 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727818+0000", 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7577999999999999e-05 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.727839+0000", 2024-09-16T04:02:23.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1123000000000001e-05 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728012+0000", 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000172366 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.728043+0000", 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1857999999999999e-05 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740357+0000", 2024-09-16T04:02:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012313996000000001 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.740427+0000", 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9789999999999994e-05 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 75/33 ECSubWrite(tid=282, reqid=client.4646.0:706, at_version=75'166, trim_to=0'0, roll_forward_to=73'164))", 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:01.036777+0000", 2024-09-16T04:02:23.207 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.093210528, 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012394228, 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.036777+0000", 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.036777+0000", 2024-09-16T04:02:23.208 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.036777+0000", 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7999999999999996e-07 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041684+0000", 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049069830000000002 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.209 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041713+0000", 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8328999999999999e-05 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041736+0000", 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3693000000000002e-05 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041890+0000", 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000153351 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.210 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.041916+0000", 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5695999999999999e-05 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.049069+0000", 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0071538490000000003 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.049171+0000", 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000101847 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 77/33 ECSubWrite(tid=307, reqid=osd.11.0:307, at_version=77'189, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.739733+0000", 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.390255031999999, 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013068305000000001, 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739733+0000", 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739733+0000", 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739734+0000", 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.677e-06 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739799+0000", 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4346999999999999e-05 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739832+0000", 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3618999999999998e-05 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739856+0000", 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3839999999999999e-05 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.740075+0000", 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021936899999999999 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.740102+0000", 2024-09-16T04:02:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6559000000000001e-05 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.752724+0000", 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01262187 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.752801+0000", 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7024e-05 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.cs6 77/33 ECSubWrite(tid=309, reqid=osd.11.0:309, at_version=77'191, trim_to=0'0, roll_forward_to=76'187))", 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.739936+0000", 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.390051389, 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015721432, 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739936+0000", 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739936+0000", 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739937+0000", 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.739992+0000", 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4749e-05 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.740020+0000", 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8279000000000001e-05 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.740042+0000", 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1773e-05 2024-09-16T04:02:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744030+0000", 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039875359999999999 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.744054+0000", 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3989e-05 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.755523+0000", 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01146983 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.755658+0000", 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013417299999999999 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s3 88/33 ECSubWrite(tid=273, reqid=osd.4.0:273, at_version=88'51, trim_to=0'0, roll_forward_to=86'49))", 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.832804+0000", 2024-09-16T04:02:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.2971831139999992, 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012635153, 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.832804+0000", 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.832804+0000", 2024-09-16T04:02:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.832805+0000", 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0599999999999999e-07 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.832878+0000", 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2855000000000006e-05 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.832912+0000", 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3729000000000002e-05 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.832935+0000", 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3016000000000001e-05 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833130+0000", 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019562299999999999 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833158+0000", 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7902000000000002e-05 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.845375+0000", 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012216411 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.845440+0000", 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4811e-05 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s3 88/33 ECSubWrite(tid=275, reqid=osd.4.0:275, at_version=88'53, trim_to=0'0, roll_forward_to=86'49))", 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.833884+0000", 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.2961037829999995, 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016504929000000002, 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833884+0000", 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833884+0000", 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.833885+0000", 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.263e-06 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.834923+0000", 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010383720000000001 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.834951+0000", 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7903e-05 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.834976+0000", 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4796000000000001e-05 2024-09-16T04:02:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.835989+0000", 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001012439 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836013+0000", 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4737e-05 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850298+0000", 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014284860999999999 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850389+0000", 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0557999999999996e-05 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 88/33 ECSubWrite(tid=207, reqid=osd.14.0:207, at_version=88'136, trim_to=0'0, roll_forward_to=86'132))", 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.836621+0000", 2024-09-16T04:02:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.2933667379999996, 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013850655, 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836621+0000", 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836621+0000", 2024-09-16T04:02:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836622+0000", 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.26e-06 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836696+0000", 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3818000000000004e-05 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836725+0000", 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9603000000000001e-05 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.836747+0000", 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1089000000000001e-05 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.839883+0000", 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031363630000000001 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.839909+0000", 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6389000000000001e-05 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850385+0000", 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010475302000000001 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850471+0000", 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6830999999999996e-05 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 92/33 ECSubWrite(tid=216, reqid=osd.14.0:216, at_version=92'143, trim_to=0'0, roll_forward_to=88'141))", 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.755741+0000", 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3742468800000003, 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014506303999999999, 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755741+0000", 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755741+0000", 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755742+0000", 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6700000000000001e-06 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755811+0000", 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755842+0000", 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1538999999999997e-05 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755909+0000", 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7150999999999995e-05 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.756063+0000", 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015385999999999999 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.756089+0000", 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5452999999999998e-05 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.768309+0000", 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012220672 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.770247+0000", 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00193765 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 92/33 ECSubWrite(tid=218, reqid=osd.14.0:218, at_version=92'145, trim_to=0'0, roll_forward_to=88'141))", 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.757499+0000", 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3724885970000003, 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012833331999999999, 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.757499+0000", 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.757499+0000", 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.757501+0000", 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.863e-06 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.757562+0000", 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1569000000000006e-05 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.758228+0000", 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00066515499999999996 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.758250+0000", 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2932999999999999e-05 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.758420+0000", 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016981900000000001 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.758446+0000", 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5743000000000001e-05 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.770258+0000", 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011812292 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.770332+0000", 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3957999999999999e-05 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s3 94/33 ECSubWrite(tid=314, reqid=osd.4.0:314, at_version=94'59, trim_to=0'0, roll_forward_to=92'57))", 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:19.752519+0000", 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.3774690010000001, 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013449449, 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.752519+0000", 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.752519+0000", 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.752519+0000", 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8999999999999996e-07 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.752577+0000", 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7892000000000001e-05 2024-09-16T04:02:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.752611+0000", 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3745999999999998e-05 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.752636+0000", 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5505999999999999e-05 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.752831+0000", 2024-09-16T04:02:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019483 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.752858+0000", 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7302000000000001e-05 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.765866+0000", 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013007708 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.765968+0000", 2024-09-16T04:02:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010177499999999999 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.0s4 94/33 ECSubWrite(tid=225, reqid=osd.14.0:225, at_version=94'151, trim_to=0'0, roll_forward_to=92'149))", 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:19.754183+0000", 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.3758050110000002, 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014255940999999999, 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.754183+0000", 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.754183+0000", 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.754183+0000", 2024-09-16T04:02:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0399999999999996e-07 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.754247+0000", 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3689000000000006e-05 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.754276+0000", 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9285000000000002e-05 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.754308+0000", 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1517000000000003e-05 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.754698+0000", 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039006400000000002 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.754726+0000", 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8347000000000002e-05 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.768344+0000", 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013618414000000001 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.768438+0000", 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4121000000000002e-05 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.3s5 94/33 ECSubWrite(tid=319, reqid=osd.13.0:319, at_version=94'76, trim_to=0'0, roll_forward_to=92'74))", 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:19.756020+0000", 2024-09-16T04:02:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.3739675870000001, 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012468056, 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.756020+0000", 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.756020+0000", 2024-09-16T04:02:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.756021+0000", 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9999999999999997e-07 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.756081+0000", 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0359000000000003e-05 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.758870+0000", 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027891230000000001 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.759027+0000", 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000156503 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.759286+0000", 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025969799999999999 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.759313+0000", 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7059e-05 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.768391+0000", 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0090776629999999997 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.768488+0000", 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7051000000000003e-05 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.257 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:23.259 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:02:23.259 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":97,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:02:22.774796+0000","last_up_change":"2024-09-16T04:00:57.414587+0000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:00:57.133339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.88243961334228516,"average_primary_affinity":0.87999999523162842,"average_primary_affinity_weighted":0.87999999523162842}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:01:05.007470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.88243961334228516,"average_primary_affinity":0.87999999523162842,"average_primary_affinity_weighted":0.87999999523162842}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-16T04:01:17.834640+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":41,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6817","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6819","nonce":3422299298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6824","nonce":3422299298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6831","nonce":3422299298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6829","nonce":3422299298}]},"public_addr":"172.21.15.37:6819/3422299298","cluster_addr":"172.21.15.37:6824/3422299298","heartbeat_back_addr":"172.21.15.37:6831/3422299298","heartbeat_front_addr":"172.21.15.37:6829/3422299298","state":["exists","up"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6809","nonce":1856047451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6811","nonce":1856047451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6815","nonce":1856047451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6813","nonce":1856047451}]},"public_addr":"172.21.15.39:6809/1856047451","cluster_addr":"172.21.15.39:6811/1856047451","heartbeat_back_addr":"172.21.15.39:6815/1856047451","heartbeat_front_addr":"172.21.15.39:6813/1856047451","state":["exists","up"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6817","nonce":3343342478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6819","nonce":3343342478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6823","nonce":3343342478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6821","nonce":3343342478}]},"public_addr":"172.21.15.117:6817/3343342478","cluster_addr":"172.21.15.117:6819/3343342478","heartbeat_back_addr":"172.21.15.117:6823/3343342478","heartbeat_front_addr":"172.21.15.117:6821/3343342478","state":["exists","up"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6803","nonce":2905616722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6807","nonce":2905616722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6816","nonce":2905616722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6811","nonce":2905616722}]},"public_addr":"172.21.15.188:6803/2905616722","cluster_addr":"172.21.15.188:6807/2905616722","heartbeat_back_addr":"172.21.15.188:6816/2905616722","heartbeat_front_addr":"172.21.15.188:6811/2905616722","state":["exists","up"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6801","nonce":2698627343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6803","nonce":2698627343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6807","nonce":2698627343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6805","nonce":2698627343}]},"public_addr":"172.21.15.37:6801/2698627343","cluster_addr":"172.21.15.37:6803/2698627343","heartbeat_back_addr":"172.21.15.37:6807/2698627343","heartbeat_front_addr":"172.21.15.37:6805/2698627343","state":["exists","up"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6825","nonce":2772028538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6827","nonce":2772028538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6831","nonce":2772028538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6829","nonce":2772028538}]},"public_addr":"172.21.15.39:6825/2772028538","cluster_addr":"172.21.15.39:6827/2772028538","heartbeat_back_addr":"172.21.15.39:6831/2772028538","heartbeat_front_addr":"172.21.15.39:6829/2772028538","state":["exists","up"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6801","nonce":3124837418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6803","nonce":3124837418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6807","nonce":3124837418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6805","nonce":3124837418}]},"public_addr":"172.21.15.117:6801/3124837418","cluster_addr":"172.21.15.117:6803/3124837418","heartbeat_back_addr":"172.21.15.117:6807/3124837418","heartbeat_front_addr":"172.21.15.117:6805/3124837418","state":["exists","up"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":1,"in":1,"weight":1,"primary_affinity":0.0573883056640625,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6806","nonce":4221150049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6810","nonce":4221150049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6815","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6817","nonce":4221150049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6813","nonce":4221150049}]},"public_addr":"172.21.15.188:6806/4221150049","cluster_addr":"172.21.15.188:6810/4221150049","heartbeat_back_addr":"172.21.15.188:6817/4221150049","heartbeat_front_addr":"172.21.15.188:6813/4221150049","state":["exists","up"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6809","nonce":3581510146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6811","nonce":3581510146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6815","nonce":3581510146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6813","nonce":3581510146}]},"public_addr":"172.21.15.37:6809/3581510146","cluster_addr":"172.21.15.37:6811/3581510146","heartbeat_back_addr":"172.21.15.37:6815/3581510146","heartbeat_front_addr":"172.21.15.37:6813/3581510146","state":["exists","up"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6801","nonce":2726119652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6803","nonce":2726119652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6807","nonce":2726119652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6805","nonce":2726119652}]},"public_addr":"172.21.15.39:6801/2726119652","cluster_addr":"172.21.15.39:6803/2726119652","heartbeat_back_addr":"172.21.15.39:6807/2726119652","heartbeat_front_addr":"172.21.15.39:6805/2726119652","state":["exists","up"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6825","nonce":737103121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6827","nonce":737103121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6831","nonce":737103121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6829","nonce":737103121}]},"public_addr":"172.21.15.117:6825/737103121","cluster_addr":"172.21.15.117:6827/737103121","heartbeat_back_addr":"172.21.15.117:6831/737103121","heartbeat_front_addr":"172.21.15.117:6829/737103121","state":["exists","up"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6823","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6824","nonce":1355328456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6828","nonce":1355328456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6833","nonce":1355328456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6831","nonce":1355328456}]},"public_addr":"172.21.15.188:6824/1355328456","cluster_addr":"172.21.15.188:6828/1355328456","heartbeat_back_addr":"172.21.15.188:6833/1355328456","heartbeat_front_addr":"172.21.15.188:6831/1355328456","state":["exists","up"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":1,"in":1,"weight":1,"primary_affinity":0.0616455078125,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6818","nonce":3693386273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6821","nonce":3693386273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6827","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6828","nonce":3693386273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6825","nonce":3693386273}]},"public_addr":"172.21.15.37:6818/3693386273","cluster_addr":"172.21.15.37:6821/3693386273","heartbeat_back_addr":"172.21.15.37:6828/3693386273","heartbeat_front_addr":"172.21.15.37:6825/3693386273","state":["exists","up"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6817","nonce":2905392027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6819","nonce":2905392027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6823","nonce":2905392027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6821","nonce":2905392027}]},"public_addr":"172.21.15.39:6817/2905392027","cluster_addr":"172.21.15.39:6819/2905392027","heartbeat_back_addr":"172.21.15.39:6823/2905392027","heartbeat_front_addr":"172.21.15.39:6821/2905392027","state":["exists","up"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6809","nonce":1375514506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6811","nonce":1375514506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6815","nonce":1375514506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6813","nonce":1375514506}]},"public_addr":"172.21.15.117:6809/1375514506","cluster_addr":"172.21.15.117:6811/1375514506","heartbeat_back_addr":"172.21.15.117:6815/1375514506","heartbeat_front_addr":"172.21.15.117:6813/1375514506","state":["exists","up"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6819","nonce":3000701187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6821","nonce":3000701187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6829","nonce":3000701187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6825","nonce":3000701187}]},"public_addr":"172.21.15.188:6819/3000701187","cluster_addr":"172.21.15.188:6821/3000701187","heartbeat_back_addr":"172.21.15.188:6829/3000701187","heartbeat_front_addr":"172.21.15.188:6825/3000701187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.245592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:05.860319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:06.602619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:58.884947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:15.931876+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:54.754303+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:56.478181+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:01.375477+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:57.934680+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:17.873726+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:56.396649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:16.811116+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.982880+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.914975+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:08.616813+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:06.605685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":17,"length":1},{"begin":30,"length":1},{"begin":35,"length":1},{"begin":38,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:02:23.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:02:23.260 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:23.256+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:23.261 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:23.273 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:23.274 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:23.274 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:23.276 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:23.276 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-16T04:02:23.360 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:23.356+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:23.434 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:23.434 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:23.434 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:02:23.434 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:23.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:02:23.461 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:23.457+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:23.465 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:23.465 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:23.465 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:23.465 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:23.465 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:23.465 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:23.465 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:23.466 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:23.466 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:23.466 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:23.466 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:23.466 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:23.469 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:23.469 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-09-16T04:02:23.562 INFO:tasks.ceph.osd.15.smithi188.stderr:2024-09-16T04:02:23.556+0000 7f731c1f6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26647) UID: 0 2024-09-16T04:02:23.565 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:02:23.568 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:02:23.575 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:23.575 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:23.575 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:02:23.575 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:02:23.575 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:23.582 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-16T04:02:23.063278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":167690803,"num_objects":155,"num_object_clones":101,"num_object_copies":1065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":11,"num_read":623,"num_read_kb":248849,"num_write":1105,"num_write_kb":170390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1549,"ondisk_log_size":1549,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":815928,"kb_used_data":293216,"kb_used_omap":24,"kb_used_meta":522279,"kb_avail":1509133512,"statfs":{"total":1546188226560,"available":1545352716288,"internally_reserved":0,"allocated":300253184,"data_stored":475004750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":534813777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8765440,"num_objects":8,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":32,"num_read_kb":23610,"num_write":33,"num_write_kb":8560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001047"},"pg_stats":[{"pgid":"3.f","version":"95'101","reported_seq":382,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.807926+0000","last_change":"2024-09-16T04:02:20.809095+0000","last_active":"2024-09-16T04:02:20.807926+0000","last_peered":"2024-09-16T04:02:20.807926+0000","last_clean":"2024-09-16T04:02:20.807926+0000","last_became_active":"2024-09-16T04:01:19.845422+0000","last_became_peered":"2024-09-16T04:01:19.845422+0000","last_unstale":"2024-09-16T04:02:20.807926+0000","last_undegraded":"2024-09-16T04:02:20.807926+0000","last_fullsized":"2024-09-16T04:02:20.807926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:35.888569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042827880999999998,"stat_sum":{"num_bytes":10682368,"num_objects":11,"num_object_clones":6,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":37,"num_read_kb":17245,"num_write":72,"num_write_kb":10960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"2"}]},{"pgid":"3.e","version":"92'22","reported_seq":299,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:02:17.755377+0000","last_change":"2024-09-16T04:02:17.756739+0000","last_active":"2024-09-16T04:02:17.755377+0000","last_peered":"2024-09-16T04:02:17.755377+0000","last_clean":"2024-09-16T04:02:17.755377+0000","last_became_active":"2024-09-16T04:01:20.266249+0000","last_became_peered":"2024-09-16T04:01:20.266249+0000","last_unstale":"2024-09-16T04:02:17.755377+0000","last_undegraded":"2024-09-16T04:02:17.755377+0000","last_fullsized":"2024-09-16T04:02:17.755377+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:41.199810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020789229999999999,"stat_sum":{"num_bytes":2703360,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":9223,"num_write":27,"num_write_kb":3040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"1"}]},{"pgid":"3.d","version":"96'130","reported_seq":430,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.791657+0000","last_change":"2024-09-16T04:02:21.791657+0000","last_active":"2024-09-16T04:02:21.791657+0000","last_peered":"2024-09-16T04:02:21.791657+0000","last_clean":"2024-09-16T04:02:21.791657+0000","last_became_active":"2024-09-16T04:02:18.755704+0000","last_became_peered":"2024-09-16T04:02:18.755704+0000","last_unstale":"2024-09-16T04:02:21.791657+0000","last_undegraded":"2024-09-16T04:02:21.791657+0000","last_fullsized":"2024-09-16T04:02:21.791657+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.048949410999999998,"stat_sum":{"num_bytes":22675456,"num_objects":17,"num_object_clones":14,"num_object_copies":119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":48,"num_read_kb":29658,"num_write":88,"num_write_kb":20176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"1"}]},{"pgid":"3.c","version":"95'304","reported_seq":570,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.819189+0000","last_change":"2024-09-16T04:02:20.820254+0000","last_active":"2024-09-16T04:02:20.819189+0000","last_peered":"2024-09-16T04:02:20.819189+0000","last_clean":"2024-09-16T04:02:20.819189+0000","last_became_active":"2024-09-16T04:01:20.259789+0000","last_became_peered":"2024-09-16T04:01:20.259789+0000","last_unstale":"2024-09-16T04:02:20.819189+0000","last_undegraded":"2024-09-16T04:02:20.819189+0000","last_fullsized":"2024-09-16T04:02:20.819189+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.423063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054567804999999997,"stat_sum":{"num_bytes":41877504,"num_objects":31,"num_object_clones":23,"num_object_copies":217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":1,"num_read":66,"num_read_kb":21526,"num_write":178,"num_write_kb":42096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"2"}]},{"pgid":"3.b","version":"95'185","reported_seq":470,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.809930+0000","last_change":"2024-09-16T04:02:20.811902+0000","last_active":"2024-09-16T04:02:20.809930+0000","last_peered":"2024-09-16T04:02:20.809930+0000","last_clean":"2024-09-16T04:02:20.809930+0000","last_became_active":"2024-09-16T04:01:20.268548+0000","last_became_peered":"2024-09-16T04:01:20.268548+0000","last_unstale":"2024-09-16T04:02:20.809930+0000","last_undegraded":"2024-09-16T04:02:20.809930+0000","last_fullsized":"2024-09-16T04:02:20.809930+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.780992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.058651123999999999,"stat_sum":{"num_bytes":21692416,"num_objects":19,"num_object_clones":13,"num_object_copies":133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":66,"num_read_kb":36935,"num_write":121,"num_write_kb":22096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"2"}]},{"pgid":"3.a","version":"95'31","reported_seq":323,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.763016+0000","last_change":"2024-09-16T04:02:21.767639+0000","last_active":"2024-09-16T04:02:21.763016+0000","last_peered":"2024-09-16T04:02:21.763016+0000","last_clean":"2024-09-16T04:02:21.763016+0000","last_became_active":"2024-09-16T04:01:20.268001+0000","last_became_peered":"2024-09-16T04:01:20.268001+0000","last_unstale":"2024-09-16T04:02:21.763016+0000","last_undegraded":"2024-09-16T04:02:21.763016+0000","last_fullsized":"2024-09-16T04:02:21.763016+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:32.829854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024843323,"stat_sum":{"num_bytes":3768320,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":8262,"num_write":11,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"1"}]},{"pgid":"3.9","version":"92'41","reported_seq":309,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:02:17.762163+0000","last_change":"2024-09-16T04:02:17.763051+0000","last_active":"2024-09-16T04:02:17.762163+0000","last_peered":"2024-09-16T04:02:17.762163+0000","last_clean":"2024-09-16T04:02:17.762163+0000","last_became_active":"2024-09-16T04:01:20.153728+0000","last_became_peered":"2024-09-16T04:01:20.153728+0000","last_unstale":"2024-09-16T04:02:17.762163+0000","last_undegraded":"2024-09-16T04:02:17.762163+0000","last_fullsized":"2024-09-16T04:02:17.762163+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:37.648803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012665793,"stat_sum":{"num_bytes":4751360,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":15943,"num_write":15,"num_write_kb":4640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"7","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"1"}]},{"pgid":"3.8","version":"95'33","reported_seq":313,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.743732+0000","last_change":"2024-09-16T04:02:21.744547+0000","last_active":"2024-09-16T04:02:21.743732+0000","last_peered":"2024-09-16T04:02:21.743732+0000","last_clean":"2024-09-16T04:02:21.743732+0000","last_became_active":"2024-09-16T04:01:20.259482+0000","last_became_peered":"2024-09-16T04:01:20.259482+0000","last_unstale":"2024-09-16T04:02:21.743732+0000","last_undegraded":"2024-09-16T04:02:21.743732+0000","last_fullsized":"2024-09-16T04:02:21.743732+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.782338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001543724,"stat_sum":{"num_bytes":3063808,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":11,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":355,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.751137+0000","last_change":"2024-09-16T04:02:16.104902+0000","last_active":"2024-09-16T04:02:21.751137+0000","last_peered":"2024-09-16T04:02:21.751137+0000","last_clean":"2024-09-16T04:02:21.751137+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:02:21.751137+0000","last_undegraded":"2024-09-16T04:02:21.751137+0000","last_fullsized":"2024-09-16T04:02:21.751137+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T04:02:16.104676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:02:16.104676+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:30.120742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":156,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.743051+0000","last_change":"2024-09-16T04:02:13.919586+0000","last_active":"2024-09-16T04:02:21.743051+0000","last_peered":"2024-09-16T04:02:21.743051+0000","last_clean":"2024-09-16T04:02:21.743051+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:02:21.743051+0000","last_undegraded":"2024-09-16T04:02:21.743051+0000","last_fullsized":"2024-09-16T04:02:21.743051+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'1","last_scrub_stamp":"2024-09-16T04:02:13.919386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:13.919386+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:39.323146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"95'54","reported_seq":347,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.754821+0000","last_change":"2024-09-16T04:02:21.755992+0000","last_active":"2024-09-16T04:02:21.754821+0000","last_peered":"2024-09-16T04:02:21.754821+0000","last_clean":"2024-09-16T04:02:21.754821+0000","last_became_active":"2024-09-16T04:01:20.258661+0000","last_became_peered":"2024-09-16T04:01:20.258661+0000","last_unstale":"2024-09-16T04:02:21.754821+0000","last_undegraded":"2024-09-16T04:02:21.754821+0000","last_fullsized":"2024-09-16T04:02:21.754821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:22.806399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001890858,"stat_sum":{"num_bytes":8355840,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":20602,"num_write":29,"num_write_kb":8736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":148,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.741686+0000","last_change":"2024-09-16T04:01:08.745182+0000","last_active":"2024-09-16T04:02:21.741686+0000","last_peered":"2024-09-16T04:02:21.741686+0000","last_clean":"2024-09-16T04:02:21.741686+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:02:21.741686+0000","last_undegraded":"2024-09-16T04:02:21.741686+0000","last_fullsized":"2024-09-16T04:02:21.741686+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:22.147268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"95'63","reported_seq":335,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.773157+0000","last_change":"2024-09-16T04:02:20.773894+0000","last_active":"2024-09-16T04:02:20.773157+0000","last_peered":"2024-09-16T04:02:20.773157+0000","last_clean":"2024-09-16T04:02:20.773157+0000","last_became_active":"2024-09-16T04:01:20.269891+0000","last_became_peered":"2024-09-16T04:01:20.269891+0000","last_unstale":"2024-09-16T04:02:20.773157+0000","last_undegraded":"2024-09-16T04:02:20.773157+0000","last_fullsized":"2024-09-16T04:02:20.773157+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.446542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020911345000000001,"stat_sum":{"num_bytes":7077888,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":14706,"num_write":58,"num_write_kb":6912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":150,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.751747+0000","last_change":"2024-09-16T04:01:08.988461+0000","last_active":"2024-09-16T04:02:21.751747+0000","last_peered":"2024-09-16T04:02:21.751747+0000","last_clean":"2024-09-16T04:02:21.751747+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:02:21.751747+0000","last_undegraded":"2024-09-16T04:02:21.751747+0000","last_fullsized":"2024-09-16T04:02:21.751747+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:27.334919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"92'149","reported_seq":403,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:02:17.785350+0000","last_change":"2024-09-16T04:02:17.786710+0000","last_active":"2024-09-16T04:02:17.785350+0000","last_peered":"2024-09-16T04:02:17.785350+0000","last_clean":"2024-09-16T04:02:17.785350+0000","last_became_active":"2024-09-16T04:01:20.257526+0000","last_became_peered":"2024-09-16T04:01:20.257526+0000","last_unstale":"2024-09-16T04:02:17.785350+0000","last_undegraded":"2024-09-16T04:02:17.785350+0000","last_fullsized":"2024-09-16T04:02:17.785350+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.143800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036410228000000003,"stat_sum":{"num_bytes":10485760,"num_objects":15,"num_object_clones":10,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":3,"num_read":35,"num_read_kb":12353,"num_write":117,"num_write_kb":13936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"7","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":150,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.742931+0000","last_change":"2024-09-16T04:01:08.746422+0000","last_active":"2024-09-16T04:02:21.742931+0000","last_peered":"2024-09-16T04:02:21.742931+0000","last_clean":"2024-09-16T04:02:21.742931+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:02:21.742931+0000","last_undegraded":"2024-09-16T04:02:21.742931+0000","last_fullsized":"2024-09-16T04:02:21.742931+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:22.147322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"96'82","reported_seq":368,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.775101+0000","last_change":"2024-09-16T04:02:21.777729+0000","last_active":"2024-09-16T04:02:21.775101+0000","last_peered":"2024-09-16T04:02:21.775101+0000","last_clean":"2024-09-16T04:02:21.775101+0000","last_became_active":"2024-09-16T04:01:19.840099+0000","last_became_peered":"2024-09-16T04:01:19.840099+0000","last_unstale":"2024-09-16T04:02:21.775101+0000","last_undegraded":"2024-09-16T04:02:21.775101+0000","last_fullsized":"2024-09-16T04:02:21.775101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:32.088812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025091113000000002,"stat_sum":{"num_bytes":9666560,"num_objects":10,"num_object_clones":7,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":23,"num_read_kb":17778,"num_write":64,"num_write_kb":7584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"1"}]},{"pgid":"3.5","version":"95'64","reported_seq":341,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.763685+0000","last_change":"2024-09-16T04:02:20.764761+0000","last_active":"2024-09-16T04:02:20.763685+0000","last_peered":"2024-09-16T04:02:20.763685+0000","last_clean":"2024-09-16T04:02:20.763685+0000","last_became_active":"2024-09-16T04:01:20.258954+0000","last_became_peered":"2024-09-16T04:01:20.258954+0000","last_unstale":"2024-09-16T04:02:20.763685+0000","last_undegraded":"2024-09-16T04:02:20.763685+0000","last_fullsized":"2024-09-16T04:02:20.763685+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.134468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016268777000000002,"stat_sum":{"num_bytes":7372800,"num_objects":8,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":5253,"num_write":40,"num_write_kb":7200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"7","length":"1"},{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":145,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.756335+0000","last_change":"2024-09-16T04:01:08.986337+0000","last_active":"2024-09-16T04:02:20.756335+0000","last_peered":"2024-09-16T04:02:20.756335+0000","last_clean":"2024-09-16T04:02:20.756335+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:02:20.756335+0000","last_undegraded":"2024-09-16T04:02:20.756335+0000","last_fullsized":"2024-09-16T04:02:20.756335+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.978331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"95'90","reported_seq":388,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.791562+0000","last_change":"2024-09-16T04:02:20.793001+0000","last_active":"2024-09-16T04:02:20.791562+0000","last_peered":"2024-09-16T04:02:20.791562+0000","last_clean":"2024-09-16T04:02:20.791562+0000","last_became_active":"2024-09-16T04:01:20.154175+0000","last_became_peered":"2024-09-16T04:01:20.154175+0000","last_unstale":"2024-09-16T04:02:20.791562+0000","last_undegraded":"2024-09-16T04:02:20.791562+0000","last_fullsized":"2024-09-16T04:02:20.791562+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:21.820333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035917245,"stat_sum":{"num_bytes":10780672,"num_objects":11,"num_object_clones":6,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":53,"num_read_kb":29970,"num_write":53,"num_write_kb":11888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7","length":"1"},{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:02:17.738972+0000","last_change":"2024-09-16T04:01:08.738000+0000","last_active":"2024-09-16T04:02:17.738972+0000","last_peered":"2024-09-16T04:02:17.738972+0000","last_clean":"2024-09-16T04:02:17.738972+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:02:17.738972+0000","last_undegraded":"2024-09-16T04:02:17.738972+0000","last_fullsized":"2024-09-16T04:02:17.738972+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:24.052853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"95'23","reported_seq":306,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.788037+0000","last_change":"2024-09-16T04:02:20.788947+0000","last_active":"2024-09-16T04:02:20.788037+0000","last_peered":"2024-09-16T04:02:20.788037+0000","last_clean":"2024-09-16T04:02:20.788037+0000","last_became_active":"2024-09-16T04:01:20.266522+0000","last_became_peered":"2024-09-16T04:01:20.266522+0000","last_unstale":"2024-09-16T04:02:20.788037+0000","last_undegraded":"2024-09-16T04:02:20.788037+0000","last_fullsized":"2024-09-16T04:02:20.788037+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:29.172070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032445757999999998,"stat_sum":{"num_bytes":1294336,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":6775,"num_write":10,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"22","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":146,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.757430+0000","last_change":"2024-09-16T04:01:08.738033+0000","last_active":"2024-09-16T04:02:20.757430+0000","last_peered":"2024-09-16T04:02:20.757430+0000","last_clean":"2024-09-16T04:02:20.757430+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:02:20.757430+0000","last_undegraded":"2024-09-16T04:02:20.757430+0000","last_fullsized":"2024-09-16T04:02:20.757430+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:34.691311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"95'17","reported_seq":299,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T04:02:21.744637+0000","last_change":"2024-09-16T04:02:21.746020+0000","last_active":"2024-09-16T04:02:21.744637+0000","last_peered":"2024-09-16T04:02:21.744637+0000","last_clean":"2024-09-16T04:02:21.744637+0000","last_became_active":"2024-09-16T04:01:20.267489+0000","last_became_peered":"2024-09-16T04:01:20.267489+0000","last_unstale":"2024-09-16T04:02:21.744637+0000","last_undegraded":"2024-09-16T04:02:21.744637+0000","last_fullsized":"2024-09-16T04:02:21.744637+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:18.791127+0000","last_clean_scrub_stamp":"2024-09-16T04:01:18.791127+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:43.427711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047560659999999998,"stat_sum":{"num_bytes":851968,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":11,"num_read_kb":2499,"num_write":6,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11","length":"1"},{"start":"1e","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":146,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T04:02:20.748522+0000","last_change":"2024-09-16T04:01:08.988772+0000","last_active":"2024-09-16T04:02:20.748522+0000","last_peered":"2024-09-16T04:02:20.748522+0000","last_clean":"2024-09-16T04:02:20.748522+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:02:20.748522+0000","last_undegraded":"2024-09-16T04:02:20.748522+0000","last_fullsized":"2024-09-16T04:02:20.748522+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:01:05.697312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:23.001594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":167100416,"num_objects":151,"num_object_clones":101,"num_object_copies":1057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":11,"num_read":487,"num_read_kb":248731,"num_write":900,"num_write_kb":166928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277303296,"data_stored":461721600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1389,"ondisk_log_size":1389,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215124,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47068,"kb_used_data":15880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324772,"statfs":{"total":96636764160,"available":96588566528,"internally_reserved":0,"allocated":16261120,"data_stored":22530116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215124,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51484,"kb_used_data":20292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320356,"statfs":{"total":96636764160,"available":96584044544,"internally_reserved":0,"allocated":20779008,"data_stored":35592260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215125,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55384,"kb_used_data":24196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316456,"statfs":{"total":96636764160,"available":96580050944,"internally_reserved":0,"allocated":24776704,"data_stored":41601092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44320,"kb_used_data":17284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327520,"statfs":{"total":96636764160,"available":96591380480,"internally_reserved":0,"allocated":17698816,"data_stored":24717688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280534,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71584,"kb_used_data":22340,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94300256,"statfs":{"total":96636764160,"available":96563462144,"internally_reserved":0,"allocated":22876160,"data_stored":34640417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56692,"kb_used_data":25496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315148,"statfs":{"total":96636764160,"available":96578711552,"internally_reserved":0,"allocated":26107904,"data_stored":43481464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280532,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37224,"kb_used_data":10188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334616,"statfs":{"total":96636764160,"available":96598646784,"internally_reserved":0,"allocated":10432512,"data_stored":14178680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280532,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46704,"kb_used_data":19676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325136,"statfs":{"total":96636764160,"available":96588939264,"internally_reserved":0,"allocated":20148224,"data_stored":32936481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280532,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44660,"kb_used_data":17624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327180,"statfs":{"total":96636764160,"available":96591032320,"internally_reserved":0,"allocated":18046976,"data_stored":31385995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38680,"kb_used_data":11644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333160,"statfs":{"total":96636764160,"available":96597155840,"internally_reserved":0,"allocated":11923456,"data_stored":17381752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215124,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47280,"kb_used_data":16084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324560,"statfs":{"total":96636764160,"available":96588349440,"internally_reserved":0,"allocated":16470016,"data_stored":23161444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62192,"kb_used_data":31000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309648,"statfs":{"total":96636764160,"available":96573079552,"internally_reserved":0,"allocated":31744000,"data_stored":52802081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247829,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50052,"kb_used_data":14696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321788,"statfs":{"total":96636764160,"available":96585510912,"internally_reserved":0,"allocated":15048704,"data_stored":20385140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247829,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43216,"kb_used_data":12020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328624,"statfs":{"total":96636764160,"available":96592510976,"internally_reserved":0,"allocated":12308480,"data_stored":20274023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":18840,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":19292160,"data_stored":33811284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247827,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47144,"kb_used_data":15956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324696,"statfs":{"total":96636764160,"available":96588488704,"internally_reserved":0,"allocated":16338944,"data_stored":26124833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30404608,"data_stored":52482048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16330752,"data_stored":23924736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14602240,"data_stored":21860352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10903552,"data_stored":19460096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":21491712,"data_stored":33837056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17891328,"data_stored":32997376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23441408,"data_stored":40828928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14954496,"data_stored":25321472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24739840,"data_stored":42688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9064448,"data_stored":13385728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":19443712,"data_stored":34820096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18763776,"data_stored":32133120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":16674816,"data_stored":30593024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13115392,"data_stored":19042304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14925824,"data_stored":21757952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10555392,"data_stored":16588800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:02:23.584 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-16T04:02:23.584 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-16T04:02:23.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:02:23.614 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:23.615 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:23.618 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:23.618 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-09-16T04:02:23.663 INFO:tasks.ceph.osd.14.smithi117.stderr:2024-09-16T04:02:23.657+0000 7f3e3878a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26403) UID: 0 2024-09-16T04:02:23.717 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 41/33 ECSubWrite(tid=70, reqid=osd.4.0:70, at_version=41'12, trim_to=0'0, roll_forward_to=40'8))", 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 56.811864550000003, 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0095027129999999994, 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.718 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901603+0000", 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901604+0000", 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.933e-06 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901660+0000", 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5979000000000001e-05 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901691+0000", 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0338999999999999e-05 2024-09-16T04:02:23.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901716+0000", 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901925+0000", 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020894899999999999 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.901950+0000", 2024-09-16T04:02:23.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5303000000000002e-05 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.911017+0000", 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0090665969999999992 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:26.911105+0000", 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8153999999999997e-05 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/33 ECSubWrite(tid=141, reqid=osd.4.0:141, at_version=66'69, trim_to=0'0, roll_forward_to=65'65))", 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:51.733946+0000", 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.979520893, 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010060861000000001, 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733946+0000", 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733946+0000", 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.733947+0000", 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.13e-06 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.734014+0000", 2024-09-16T04:02:23.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6920000000000003e-05 2024-09-16T04:02:23.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.734038+0000", 2024-09-16T04:02:23.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4213e-05 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.734058+0000", 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0012000000000001e-05 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.735934+0000", 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001875418 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.735958+0000", 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4589000000000001e-05 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.743961+0000", 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0080020620000000008 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:51.744007+0000", 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6517000000000002e-05 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.727 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 77/33 ECSubWrite(tid=195, reqid=osd.4.0:195, at_version=77'102, trim_to=0'0, roll_forward_to=76'98))", 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:02.734821+0000", 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 20.978645959000001, 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0089874410000000005, 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734821+0000", 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734821+0000", 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734823+0000", 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734891+0000", 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7774000000000005e-05 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734920+0000", 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9549000000000001e-05 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.734946+0000", 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5732999999999999e-05 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.735341+0000", 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039543 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.735365+0000", 2024-09-16T04:02:23.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3543000000000001e-05 2024-09-16T04:02:23.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.743728+0000", 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0083628179999999993 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:02.743809+0000", 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0900000000000001e-05 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 82/33 ECSubWrite(tid=231, reqid=osd.4.0:231, at_version=82'37, trim_to=0'0, roll_forward_to=81'35))", 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:07.774088+0000", 2024-09-16T04:02:23.733 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.939379098, 2024-09-16T04:02:23.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012278206999999999, 2024-09-16T04:02:23.734 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.734 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.734 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.734 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774088+0000", 2024-09-16T04:02:23.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774088+0000", 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774088+0000", 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5400000000000002e-07 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774135+0000", 2024-09-16T04:02:23.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6492000000000001e-05 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774169+0000", 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3735000000000001e-05 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774193+0000", 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4243000000000002e-05 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774420+0000", 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022687099999999999 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774447+0000", 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7659000000000001e-05 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.786286+0000", 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011838768 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.786366+0000", 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9985000000000001e-05 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.739 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 82/33 ECSubWrite(tid=232, reqid=osd.4.0:232, at_version=82'125, trim_to=0'0, roll_forward_to=81'123))", 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:07.774243+0000", 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.939224214999999, 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010593569000000001, 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774243+0000", 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774243+0000", 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774244+0000", 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8000000000000004e-07 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774293+0000", 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9394999999999997e-05 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774317+0000", 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3822999999999999e-05 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774339+0000", 2024-09-16T04:02:23.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2476e-05 2024-09-16T04:02:23.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.743 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.743 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774525+0000", 2024-09-16T04:02:23.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018559100000000001 2024-09-16T04:02:23.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774552+0000", 2024-09-16T04:02:23.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6656e-05 2024-09-16T04:02:23.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.784724+0000", 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010171918 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.784836+0000", 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011283 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.745 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 82/33 ECSubWrite(tid=233, reqid=osd.4.0:233, at_version=82'39, trim_to=0'0, roll_forward_to=81'35))", 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:07.774398+0000", 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.939068853, 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013207004, 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774398+0000", 2024-09-16T04:02:23.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774398+0000", 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774399+0000", 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1960000000000001e-06 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774458+0000", 2024-09-16T04:02:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8782000000000001e-05 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774492+0000", 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3995999999999998e-05 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774518+0000", 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5686000000000001e-05 2024-09-16T04:02:23.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778718+0000", 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042003500000000003 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.778733+0000", 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4856e-05 2024-09-16T04:02:23.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.787517+0000", 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0087838010000000008 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.787605+0000", 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8337000000000005e-05 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.751 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 82/33 ECSubWrite(tid=234, reqid=osd.4.0:234, at_version=82'127, trim_to=0'0, roll_forward_to=81'123))", 2024-09-16T04:02:23.751 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:07.774575+0000", 2024-09-16T04:02:23.751 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.938891735, 2024-09-16T04:02:23.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012262643, 2024-09-16T04:02:23.751 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774575+0000", 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774575+0000", 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774576+0000", 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.043e-06 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774636+0000", 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9842000000000001e-05 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774666+0000", 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.957e-05 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.774685+0000", 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9488999999999999e-05 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.777967+0000", 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032816719999999998 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.777990+0000", 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3070000000000001e-05 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.786796+0000", 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0088055370000000004 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:07.786838+0000", 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2419999999999997e-05 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.756 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 88/33 ECSubWrite(tid=273, reqid=osd.4.0:273, at_version=88'51, trim_to=0'0, roll_forward_to=86'49))", 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.840629+0000", 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.8728378630000009, 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0094140839999999996, 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840629+0000", 2024-09-16T04:02:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840629+0000", 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840632+0000", 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5840000000000002e-06 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840720+0000", 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7848000000000003e-05 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840768+0000", 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8415000000000003e-05 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.840792+0000", 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4423e-05 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841005+0000", 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00021258799999999999 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841032+0000", 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6818999999999999e-05 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.849947+0000", 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0089147029999999995 2024-09-16T04:02:23.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.850043+0000", 2024-09-16T04:02:23.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6704000000000001e-05 2024-09-16T04:02:23.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.761 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 88/33 ECSubWrite(tid=275, reqid=osd.4.0:275, at_version=88'53, trim_to=0'0, roll_forward_to=86'49))", 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.841050+0000", 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.8724175689999996, 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014310804, 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.762 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841050+0000", 2024-09-16T04:02:23.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841050+0000", 2024-09-16T04:02:23.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841051+0000", 2024-09-16T04:02:23.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.043e-06 2024-09-16T04:02:23.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841110+0000", 2024-09-16T04:02:23.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9731999999999997e-05 2024-09-16T04:02:23.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841161+0000", 2024-09-16T04:02:23.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0881999999999998e-05 2024-09-16T04:02:23.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841180+0000", 2024-09-16T04:02:23.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9266000000000002e-05 2024-09-16T04:02:23.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842553+0000", 2024-09-16T04:02:23.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013721670000000001 2024-09-16T04:02:23.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842577+0000", 2024-09-16T04:02:23.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4850000000000001e-05 2024-09-16T04:02:23.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.855250+0000", 2024-09-16T04:02:23.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0126727 2024-09-16T04:02:23.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.855360+0000", 2024-09-16T04:02:23.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000110164 2024-09-16T04:02:23.772 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.772 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.772 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.772 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 88/33 ECSubWrite(tid=276, reqid=client.4646.0:903, at_version=88'149, trim_to=0'0, roll_forward_to=86'143))", 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:13.841227+0000", 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.8722404739999998, 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016201380000000001, 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841227+0000", 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841227+0000", 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841228+0000", 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2270000000000001e-06 2024-09-16T04:02:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841425+0000", 2024-09-16T04:02:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019756900000000001 2024-09-16T04:02:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841453+0000", 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7288999999999999e-05 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.841473+0000", 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0103e-05 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842142+0000", 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00066884699999999997 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.842167+0000", 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4902999999999999e-05 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.857337+0000", 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015170301000000001 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:13.857428+0000", 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1141e-05 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 92/33 ECSubWrite(tid=295, reqid=osd.4.0:295, at_version=92'159, trim_to=0'0, roll_forward_to=91'157))", 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.735866+0000", 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.9776013040000002, 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0094861470000000003, 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.735866+0000", 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.735866+0000", 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.735866+0000", 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3600000000000004e-07 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.735927+0000", 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1152e-05 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.735954+0000", 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6625999999999999e-05 2024-09-16T04:02:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.735983+0000", 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8829000000000001e-05 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736140+0000", 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000156704 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736167+0000", 2024-09-16T04:02:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7106000000000001e-05 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.745114+0000", 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0089471870000000005 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.745352+0000", 2024-09-16T04:02:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023800700000000001 2024-09-16T04:02:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 92/33 ECSubWrite(tid=294, reqid=osd.4.0:294, at_version=92'55, trim_to=0'0, roll_forward_to=88'53))", 2024-09-16T04:02:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.736493+0000", 2024-09-16T04:02:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.9769739360000003, 2024-09-16T04:02:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0091536859999999994, 2024-09-16T04:02:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736493+0000", 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736493+0000", 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736495+0000", 2024-09-16T04:02:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.787e-06 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736556+0000", 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0862000000000001e-05 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736584+0000", 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8002999999999999e-05 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736659+0000", 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4771999999999995e-05 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736969+0000", 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000310606 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736994+0000", 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5082000000000001e-05 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.745566+0000", 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0085715889999999993 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.745647+0000", 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0984999999999998e-05 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 92/33 ECSubWrite(tid=296, reqid=osd.4.0:296, at_version=92'161, trim_to=0'0, roll_forward_to=91'157))", 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.736724+0000", 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.9767426409999995, 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010248963999999999, 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736724+0000", 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736724+0000", 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736725+0000", 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.47e-07 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736823+0000", 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7937999999999996e-05 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736853+0000", 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0114999999999999e-05 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.736874+0000", 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.039e-05 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.738483+0000", 2024-09-16T04:02:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001609213 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.738514+0000", 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.129e-05 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.746890+0000", 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0083755759999999992 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.746973+0000", 2024-09-16T04:02:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3494999999999999e-05 2024-09-16T04:02:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.es1 92/33 ECSubWrite(tid=39, reqid=osd.2.0:39, at_version=92'22, trim_to=0'0, roll_forward_to=88'20))", 2024-09-16T04:02:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.737493+0000", 2024-09-16T04:02:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.9759738579999997, 2024-09-16T04:02:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010412786, 2024-09-16T04:02:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737493+0000", 2024-09-16T04:02:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737493+0000", 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737494+0000", 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5000000000000003e-07 2024-09-16T04:02:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737547+0000", 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3171999999999998e-05 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737578+0000", 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1486000000000003e-05 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737601+0000", 2024-09-16T04:02:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2481999999999999e-05 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737931+0000", 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00032960699999999999 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.737956+0000", 2024-09-16T04:02:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5592000000000002e-05 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.747851+0000", 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0098948950000000008 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.747906+0000", 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5002e-05 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 92/33 ECSubWrite(tid=297, reqid=osd.4.0:297, at_version=92'57, trim_to=0'0, roll_forward_to=88'53))", 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.752617+0000", 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.9608498919999997, 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0092393430000000006, 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.752617+0000", 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.752617+0000", 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.752625+0000", 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.96e-06 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.752676+0000", 2024-09-16T04:02:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1186000000000001e-05 2024-09-16T04:02:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.752728+0000", 2024-09-16T04:02:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1542000000000002e-05 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.752752+0000", 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4536e-05 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.752910+0000", 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000157837 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.752937+0000", 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6545000000000001e-05 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.761766+0000", 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0088289230000000007 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.761857+0000", 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0814000000000002e-05 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 92/33 ECSubWrite(tid=299, reqid=osd.4.0:299, at_version=92'163, trim_to=0'0, roll_forward_to=92'161))", 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.755073+0000", 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.958393761, 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010763616, 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755073+0000", 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755073+0000", 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755076+0000", 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2900000000000001e-06 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755160+0000", 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4667999999999996e-05 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755245+0000", 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4264999999999994e-05 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755270+0000", 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5023000000000001e-05 2024-09-16T04:02:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755604+0000", 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000334552 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755629+0000", 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4736000000000002e-05 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.765715+0000", 2024-09-16T04:02:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010086061 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.765837+0000", 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122021 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 92/33 ECSubWrite(tid=300, reqid=osd.4.0:300, at_version=92'165, trim_to=0'0, roll_forward_to=92'161))", 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.755966+0000", 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.9575014780000002, 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013824316, 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755966+0000", 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755966+0000", 2024-09-16T04:02:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.755968+0000", 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9369999999999998e-06 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.756021+0000", 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3161999999999997e-05 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.756051+0000", 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0035999999999998e-05 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.756082+0000", 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1343000000000001e-05 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.760203+0000", 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041207359999999998 2024-09-16T04:02:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.760229+0000", 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6029000000000001e-05 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.769670+0000", 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0094407029999999999 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.769790+0000", 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012037 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.1s2 94/33 ECSubWrite(tid=314, reqid=osd.4.0:314, at_version=94'59, trim_to=0'0, roll_forward_to=92'57))", 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:19.757038+0000", 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.9564292989999998, 2024-09-16T04:02:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0090654480000000003, 2024-09-16T04:02:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757038+0000", 2024-09-16T04:02:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757038+0000", 2024-09-16T04:02:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757039+0000", 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0640000000000001e-06 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757093+0000", 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3885000000000003e-05 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757125+0000", 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1885999999999999e-05 2024-09-16T04:02:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757145+0000", 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9959999999999999e-05 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757333+0000", 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018821899999999999 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757359+0000", 2024-09-16T04:02:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5896000000000001e-05 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.765972+0000", 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0086128309999999996 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.766103+0000", 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131707 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.bs6 94/33 ECSubWrite(tid=315, reqid=osd.4.0:315, at_version=94'173, trim_to=0'0, roll_forward_to=93'169))", 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:19.757201+0000", 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.9562660279999999, 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010833438000000001, 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757201+0000", 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757201+0000", 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757202+0000", 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0899999999999999e-06 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757278+0000", 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6038000000000006e-05 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757308+0000", 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0105000000000001e-05 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.757328+0000", 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0083999999999999e-05 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.758585+0000", 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012569790000000001 2024-09-16T04:02:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.758614+0000", 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8895999999999999e-05 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.767925+0000", 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0093106249999999995 2024-09-16T04:02:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.768034+0000", 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109621 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.es1 94/33 ECSubWrite(tid=46, reqid=osd.2.0:46, at_version=94'26, trim_to=0'0, roll_forward_to=93'24))", 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:19.758822+0000", 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.9546451830000002, 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0092325759999999993, 2024-09-16T04:02:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.758822+0000", 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.758822+0000", 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.758823+0000", 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9599999999999998e-07 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.758955+0000", 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000132461 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.759576+0000", 2024-09-16T04:02:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062133299999999995 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.759606+0000", 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9476e-05 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.760150+0000", 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00054366500000000003 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.760175+0000", 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5863e-05 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.767949+0000", 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0077740079999999998 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:19.768054+0000", 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000104974 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:23.831 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:23.833 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:02:23.833 INFO:tasks.ceph.osd.9.smithi039.stderr:2024-09-16T04:02:23.828+0000 7f5850a78640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26264) UID: 0 2024-09-16T04:02:23.833 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:23.834 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:23.834 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:23.835 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-09-16T04:02:23.933 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-16T04:02:23.928+0000 7fb40e296640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26259) UID: 0 2024-09-16T04:02:23.989 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:23.990 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:23.993 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:23.993 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-09-16T04:02:24.010 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:24.010 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:24.010 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-16T04:02:24.010 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:24.024 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:02:24.034 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:24.033+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:24.135 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-09-16T04:02:24.131+0000 7fd77bec0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26373) UID: 0 2024-09-16T04:02:24.142 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:24.143 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:24.145 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:24.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 'trim stale osdmaps' 2024-09-16T04:02:24.190 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:24.190 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-16T04:02:24.190 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-16T04:02:24.191 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-16T04:02:24.191 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:24.205 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:02:24.236 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:24.233+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:24.332 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-16T04:02:24.332 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-16T04:02:24.332 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-16T04:02:24.332 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-16T04:02:24.332 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.332 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:24.332 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 85.866541978000001, 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.56051925899999999, 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:02:24.333 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460471+0000", 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460473+0000", 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.097e-06 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.334 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460488+0000", 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5335000000000001e-05 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460512+0000", 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4108999999999999e-05 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460533+0000", 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0721999999999999e-05 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.335 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460655+0000", 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122328 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:58.460689+0000", 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014402+0000", 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.55371241699999996 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.336 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.014725+0000", 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000323345 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.015144+0000", 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041881500000000001 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020086+0000", 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049418869999999998 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.337 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020836+0000", 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074984300000000004 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020926+0000", 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0070999999999999e-05 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.020990+0000", 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4256000000000003e-05 2024-09-16T04:02:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 84.914273000999998, 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019211259000000001, 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:24.339 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 117 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412740+0000", 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.340 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412741+0000", 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8700000000000005e-07 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412747+0000", 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1929999999999998e-06 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412757+0000", 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0039e-05 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.341 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.412764+0000", 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3259999999999998e-06 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426386+0000", 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013621668999999999 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426547+0000", 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000161192 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.342 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.426753+0000", 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00020624599999999999 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.428173+0000", 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001419305 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431799+0000", 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036262320000000001 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431890+0000", 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0987000000000001e-05 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.431951+0000", 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1383000000000005e-05 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 84.561410534000004, 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017211085000000001, 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 161 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.345 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765602+0000", 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765603+0000", 2024-09-16T04:02:24.346 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765727+0000", 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124184 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765750+0000", 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2296000000000001e-05 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765759+0000", 2024-09-16T04:02:24.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2760000000000006e-06 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775766+0000", 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010007085000000001 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.775856+0000", 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9683000000000002e-05 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.776003+0000", 2024-09-16T04:02:24.348 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014695999999999999 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779325+0000", 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033221930000000002 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782596+0000", 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032713519999999999 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.349 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782726+0000", 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013011500000000001 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782813+0000", 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7174000000000003e-05 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:02:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 84.561236132000005, 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017217072999999999, 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 162 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765777+0000", 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765778+0000", 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765784+0000", 2024-09-16T04:02:24.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2659999999999998e-06 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765794+0000", 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.029e-05 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765809+0000", 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4474999999999999e-05 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777438+0000", 2024-09-16T04:02:24.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011628929 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777529+0000", 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1577999999999996e-05 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.777666+0000", 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013660699999999999 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781937+0000", 2024-09-16T04:02:24.354 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042713660000000004 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782885+0000", 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00094829699999999996 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782947+0000", 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1261999999999997e-05 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.782994+0000", 2024-09-16T04:02:24.355 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7139999999999999e-05 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 84.561185749000003, 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01977027, 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:24.356 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 163 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.357 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765827+0000", 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765828+0000", 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765834+0000", 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7830000000000004e-06 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.358 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765844+0000", 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0149000000000001e-05 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765852+0000", 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0199999999999994e-06 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778628+0000", 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012776183999999999 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778715+0000", 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6941000000000007e-05 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.778850+0000", 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013517800000000001 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783007+0000", 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041571960000000002 2024-09-16T04:02:24.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785458+0000", 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002450626 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785539+0000", 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1213999999999999e-05 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.361 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785597+0000", 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8258999999999999e-05 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 84.561140072000001, 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.019887134000000001, 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.362 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 164 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765873+0000", 2024-09-16T04:02:24.363 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765874+0000", 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0130000000000001e-06 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.765996+0000", 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122039 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.364 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766015+0000", 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9405000000000001e-05 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766025+0000", 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0096e-05 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779427+0000", 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013401569 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.779649+0000", 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022229500000000001 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.780098+0000", 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00044856699999999998 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.366 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783018+0000", 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002919792 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785656+0000", 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026383489999999999 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785714+0000", 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8177000000000002e-05 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.785760+0000", 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5831999999999998e-05 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4258.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 84.560969150000005, 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021760347999999999, 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4258", 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/2444120344", 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 165 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.369 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766044+0000", 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766045+0000", 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766051+0000", 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8140000000000003e-06 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766062+0000", 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0592e-05 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.766069+0000", 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3629999999999996e-06 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781249+0000", 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015179349999999999 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781356+0000", 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000107747 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.781496+0000", 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013985500000000001 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.783295+0000", 2024-09-16T04:02:24.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017986199999999999 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787679+0000", 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0043837559999999999 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787752+0000", 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3192e-05 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:00:59.787804+0000", 2024-09-16T04:02:24.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2182999999999998e-05 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:347 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.4040319, 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018664877, 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.374 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 347 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922981+0000", 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922982+0000", 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.457e-06 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.922991+0000", 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923032+0000", 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0878999999999999e-05 2024-09-16T04:02:24.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923051+0000", 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9406e-05 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923225+0000", 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000173781 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923244+0000", 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8583000000000001e-05 2024-09-16T04:02:24.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928577+0000", 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053332600000000003 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.928836+0000", 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025873200000000002 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.941646+0000", 2024-09-16T04:02:24.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.01281 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:348 3.2s0 3:45f0f92f:::smithi18831615-17:head [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.403898720999997, 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020600073999999999, 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.379 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 348 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923114+0000", 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923116+0000", 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8530000000000001e-06 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923127+0000", 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1396000000000001e-05 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.384 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923152+0000", 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4253e-05 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923168+0000", 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6685999999999999e-05 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923332+0000", 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016361899999999999 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.923353+0000", 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0894999999999999e-05 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.929889+0000", 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0065358930000000001 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.930145+0000", 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000256452 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:29.943714+0000", 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013569027000000001 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:477 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.585073721999997, 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017902273, 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.387 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 477 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741939+0000", 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741942+0000", 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8830000000000002e-06 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741957+0000", 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4922999999999999e-05 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.741991+0000", 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4094999999999998e-05 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742009+0000", 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7958999999999999e-05 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742143+0000", 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013377699999999999 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.390 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742155+0000", 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2692e-05 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.747863+0000", 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057079360000000003 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.748115+0000", 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025199499999999998 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.759841+0000", 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011726013 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:478 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.584962150999999, 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020150529, 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.392 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 478 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742051+0000", 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742052+0000", 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.423e-06 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742061+0000", 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8370000000000003e-06 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742084+0000", 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3034999999999999e-05 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742100+0000", 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5798999999999999e-05 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742319+0000", 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000218789 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742340+0000", 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1069000000000001e-05 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.750020+0000", 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0076799820000000001 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.750277+0000", 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025761000000000002 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.762201+0000", 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011923985 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:479 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.584880134999999, 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021828410999999999, 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.397 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 479 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742133+0000", 2024-09-16T04:02:24.398 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742134+0000", 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2929999999999999e-06 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742143+0000", 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7569999999999993e-06 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742163+0000", 2024-09-16T04:02:24.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9839000000000001e-05 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742176+0000", 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3782e-05 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742460+0000", 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00028378600000000001 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:24.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.742471+0000", 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0856000000000001e-05 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.753628+0000", 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011156981999999999 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.753886+0000", 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00025786400000000001 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:46.763961+0000", 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010075252 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:612 3.3s0 3:c7e4f386:::smithi18831615-44:head [append 0~655360] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.917245+0000", 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.409768314000001, 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020015294999999999, 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.402 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 612 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.917245+0000", 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.917245+0000", 2024-09-16T04:02:24.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.917248+0000", 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2789999999999999e-06 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918441+0000", 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011929919999999999 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918481+0000", 2024-09-16T04:02:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0342000000000001e-05 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918499+0000", 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7396000000000001e-05 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918669+0000", 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017088699999999999 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.918683+0000", 2024-09-16T04:02:24.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3613e-05 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.923229+0000", 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045464520000000003 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.923402+0000", 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000172786 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.925580+0000", 2024-09-16T04:02:24.406 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021778050000000001 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.936521+0000", 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010940938000000001 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.937081+0000", 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00055951899999999997 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.407 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.937260+0000", 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017928599999999999 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:614 3.3s0 3:c7e4f386:::smithi18831615-44:head [append 0~475136] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.926452+0000", 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.400560691999999, 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017618722999999999, 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.408 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 614 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.926452+0000", 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.926452+0000", 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.409 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.926454+0000", 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9700000000000002e-06 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927237+0000", 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078264100000000004 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927270+0000", 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3569000000000003e-05 2024-09-16T04:02:24.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927293+0000", 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2578999999999998e-05 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.932624+0000", 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053313350000000004 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.932929+0000", 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000304739 2024-09-16T04:02:24.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.935075+0000", 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021457989999999999 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.938167+0000", 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00309254 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.943845+0000", 2024-09-16T04:02:24.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0056773860000000004 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.944071+0000", 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022616499999999999 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:615 3.3s0 3:c7e4f386:::smithi18831615-44:head [append 0~81920] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.927365+0000", 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.399648345999999, 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018348695000000002, 2024-09-16T04:02:24.413 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 615 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927365+0000", 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.414 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927365+0000", 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927367+0000", 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.103e-06 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927547+0000", 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018066999999999999 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927581+0000", 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3782000000000003e-05 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927599+0000", 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8133000000000001e-05 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.938506+0000", 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010906735000000001 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.938732+0000", 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022580500000000001 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.939846+0000", 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011143679999999999 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.944308+0000", 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044614650000000004 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.945525+0000", 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012174270000000001 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.945713+0000", 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000188207 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:616 3.3s0 3:c7e4f386:::smithi18831615-44:head [setxattr _header (57) in=7b] snapc 19=[19,18,17,16,15,14,13,11,10,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:24.418 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.927636+0000", 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.399376739000001, 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021030818999999999, 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 616 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.419 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927636+0000", 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927636+0000", 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927638+0000", 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.420 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927648+0000", 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8630000000000003e-06 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927669+0000", 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1563e-05 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927689+0000", 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9998999999999999e-05 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.421 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.940931+0000", 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013242017 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.941083+0000", 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015142700000000001 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.942463+0000", 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001380582 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.944932+0000", 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0024683270000000002 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.948416+0000", 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034847419999999999 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.948667+0000", 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000250633 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.423 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:617 3.3s0 3:c7e4f386:::smithi18831615-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:01:55.927718+0000", 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.399295175999999, 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026128365000000001, 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 617 2024-09-16T04:02:24.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927718+0000", 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927718+0000", 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927719+0000", 2024-09-16T04:02:24.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4839999999999999e-06 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927727+0000", 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5989999999999996e-06 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927752+0000", 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5398999999999999e-05 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.927772+0000", 2024-09-16T04:02:24.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0256000000000001e-05 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.946010+0000", 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018237386000000001 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for rw locks", 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.946244+0000", 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023392399999999999 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.949754+0000", 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035106040000000001 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.949954+0000", 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000199913 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:01:55.953846+0000", 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0038917999999999999 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.428 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:705 3.2s0 3:45f0f92f:::smithi18831615-17:11 [read 0~2342912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:01.000458+0000", 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.326555166999999, 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018396058, 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 705 2024-09-16T04:02:24.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000458+0000", 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000458+0000", 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000459+0000", 2024-09-16T04:02:24.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000468+0000", 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0529999999999996e-06 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000490+0000", 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2382999999999999e-05 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000505+0000", 2024-09-16T04:02:24.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5009e-05 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000914+0000", 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000408737 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.000933+0000", 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9286000000000001e-05 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.006979+0000", 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060450290000000004 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.007138+0000", 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015961400000000001 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:01.018854+0000", 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011715621000000001 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4646.0:839 3.2s0 3:45f0f92f:::smithi18831615-17:head [append 0~688128] snapc 22=[22,21,20,1f,1e,1d,1c,1b,19,18,17,16,15,11,f,e,d,a,7] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:08.797097+0000", 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.529916126, 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017363775000000001, 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4646", 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 839 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.797097+0000", 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.797097+0000", 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.797100+0000", 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1030000000000002e-06 2024-09-16T04:02:24.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.799104+0000", 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0020044350000000002 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.799142+0000", 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8232000000000003e-05 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.799159+0000", 2024-09-16T04:02:24.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6906e-05 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.799302+0000", 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014293899999999999 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.799482+0000", 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00018001900000000001 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.801744+0000", 2024-09-16T04:02:24.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022618769999999998 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.809290+0000", 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0075454739999999999 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.814287+0000", 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0049976580000000003 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:08.814460+0000", 2024-09-16T04:02:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000173132 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDECSubOpWrite(3.4s6 92/33 ECSubWrite(tid=112, reqid=osd.8.0:112, at_version=92'76, trim_to=0'0, roll_forward_to=89'72))", 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-16T04:02:17.746839+0000", 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.5801741719999995, 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018400566, 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.746839+0000", 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.746839+0000", 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.746841+0000", 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2000000000000001e-06 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.746907+0000", 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6029000000000001e-05 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.746955+0000", 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7741999999999998e-05 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.746982+0000", 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6939000000000001e-05 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.751665+0000", 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046835870000000003 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.751692+0000", 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6608999999999999e-05 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.765125+0000", 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013433142 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-16T04:02:24.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-16T04:02:17.765239+0000", 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000114318 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-16T04:02:24.443 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-16T04:02:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:24.445 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:24.445 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:24.445 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:24.445 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:24.446 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:24.446 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:24.446 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:24.446 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:24.446 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:24.446 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:24.446 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:24.446 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:24.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:02:24.447 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:24.440+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:24.447 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:24.447 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 'trim stale osdmaps' 2024-09-16T04:02:24.545 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-09-16T04:02:24.541+0000 7ffb19028640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26399) UID: 0 2024-09-16T04:02:24.579 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:24.579 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:24.579 INFO:teuthology.orchestra.run.smithi117.stdout: "num_ops": 0 2024-09-16T04:02:24.579 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:24.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:02:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:24.609 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:24.612 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:24.612 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 'trim stale osdmaps' 2024-09-16T04:02:24.646 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-16T04:02:24.644+0000 7fe36ff63640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26262) UID: 0 2024-09-16T04:02:24.747 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-09-16T04:02:24.743+0000 7fb363d86640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26377) UID: 0 2024-09-16T04:02:24.755 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:24.755 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [], 2024-09-16T04:02:24.755 INFO:teuthology.orchestra.run.smithi117.stdout: "complaint_time": 30, 2024-09-16T04:02:24.756 INFO:teuthology.orchestra.run.smithi117.stdout: "num_blocked_ops": 0 2024-09-16T04:02:24.756 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:24.760 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:24.760 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:24.761 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:24.761 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:24.761 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:24.761 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:24.762 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:24.762 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:24.762 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:24.762 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:24.762 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:24.762 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:24.763 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:24.764 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:24.764 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 'trim stale osdmaps' 2024-09-16T04:02:24.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:02:24.795 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:02:24.804 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'unique_pool_0' removed 2024-09-16T04:02:24.811 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:02:24.835 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T04:02:24.845 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "size": 20, 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 600, 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "ops": [ 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:608 3.es0 3:731b4600:::smithi18831615-23:head [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:01:55.898198+0000", 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 28.998676084, 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012172129, 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.901 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 608 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898198+0000", 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.902 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898198+0000", 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898201+0000", 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4499999999999998e-06 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898214+0000", 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3733999999999999e-05 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.903 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898243+0000", 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.8544999999999999e-05 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898261+0000", 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8349999999999999e-05 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898441+0000", 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000179882 2024-09-16T04:02:24.904 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.898451+0000", 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.8730000000000002e-06 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.903831+0000", 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0053801359999999998 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:24.905 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.904520+0000", 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00068860700000000004 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:01:55.910370+0000", 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0058505520000000002 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:765 3.es0 3:74948e3e:::smithi18831615-25:head [read 0~1638400,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:01.942890+0000", 2024-09-16T04:02:24.906 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 22.953984728999998, 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015336928999999999, 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 765 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942890+0000", 2024-09-16T04:02:24.907 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942890+0000", 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942891+0000", 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.297e-06 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.908 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942898+0000", 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.4229999999999999e-06 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942916+0000", 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7516e-05 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.942930+0000", 2024-09-16T04:02:24.909 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3845999999999999e-05 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.952436+0000", 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0095067669999999993 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.952686+0000", 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00024933099999999999 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.910 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:01.958226+0000", 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0055407490000000002 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.4s1 81/33 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=81'50, trim_to=0'0, roll_forward_to=80'48))", 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.735794+0000", 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.161080788, 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.025016644000000001, 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.911 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735794+0000", 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735794+0000", 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.912 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735795+0000", 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7060000000000001e-06 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735854+0000", 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8462999999999999e-05 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735886+0000", 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2058999999999999e-05 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.913 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.735908+0000", 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2353000000000001e-05 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738447+0000", 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0025387460000000001 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738467+0000", 2024-09-16T04:02:24.914 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.0635000000000001e-05 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.760710+0000", 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.022242893999999999 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.760810+0000", 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.9788000000000001e-05 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.915 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.4s1 81/33 ECSubWrite(tid=82, reqid=osd.8.0:82, at_version=81'52, trim_to=0'0, roll_forward_to=80'48))", 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.738367+0000", 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.15850717, 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.032503008, 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.916 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738367+0000", 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738367+0000", 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738368+0000", 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.2500000000000001e-06 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738418+0000", 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.9885000000000001e-05 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738450+0000", 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1633e-05 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.738517+0000", 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7151999999999997e-05 2024-09-16T04:02:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740733+0000", 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0022160420000000001 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740757+0000", 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3750000000000001e-05 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770786+0000", 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.030029160999999999 2024-09-16T04:02:24.919 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770870+0000", 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.4135000000000007e-05 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs1 81/33 ECSubWrite(tid=334, reqid=osd.11.0:334, at_version=81'68, trim_to=0'0, roll_forward_to=77'66))", 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.739676+0000", 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.157197890999999, 2024-09-16T04:02:24.920 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.028451536999999999, 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739676+0000", 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739676+0000", 2024-09-16T04:02:24.921 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739678+0000", 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.916e-06 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739737+0000", 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.8659000000000002e-05 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739789+0000", 2024-09-16T04:02:24.922 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.1949e-05 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.739812+0000", 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3553e-05 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740209+0000", 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000396463 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.740234+0000", 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5026000000000001e-05 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768051+0000", 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.027817172000000001 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768128+0000", 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6798999999999994e-05 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.3s4 81/33 ECSubWrite(tid=281, reqid=osd.13.0:281, at_version=81'59, trim_to=0'0, roll_forward_to=77'57))", 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.741148+0000", 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.155726268999999, 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.026862554, 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741148+0000", 2024-09-16T04:02:24.925 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741148+0000", 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741149+0000", 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.4300000000000003e-07 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.926 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741201+0000", 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2519000000000003e-05 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741225+0000", 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4247e-05 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741247+0000", 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1149e-05 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.927 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741399+0000", 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00015270400000000001 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.741423+0000", 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4070000000000002e-05 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.928 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.767907+0000", 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.026483754000000002 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768011+0000", 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010346800000000001 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.3s4 81/33 ECSubWrite(tid=282, reqid=osd.13.0:282, at_version=81'61, trim_to=0'0, roll_forward_to=77'57))", 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.742752+0000", 2024-09-16T04:02:24.929 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.154122141999999, 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.028545241999999998, 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742752+0000", 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.930 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742752+0000", 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742754+0000", 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7400000000000001e-06 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742807+0000", 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2961999999999999e-05 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.931 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742849+0000", 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.2329000000000001e-05 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.742873+0000", 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3883000000000002e-05 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755109+0000", 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012235721 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.932 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755134+0000", 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4805999999999999e-05 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771234+0000", 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.016100224999999999 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771297+0000", 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.3576000000000003e-05 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.933 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 81/33 ECSubWrite(tid=155, reqid=osd.14.0:155, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.747358+0000", 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.149516741999999, 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020922782000000001, 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.934 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747358+0000", 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747358+0000", 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747358+0000", 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.935 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747413+0000", 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.4619e-05 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747442+0000", 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9422000000000001e-05 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747464+0000", 2024-09-16T04:02:24.936 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.179e-05 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747633+0000", 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00016887 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.747668+0000", 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.5203000000000002e-05 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.937 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768152+0000", 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.020483627000000001 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.768280+0000", 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00012855399999999999 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.0s1 81/33 ECSubWrite(tid=156, reqid=osd.14.0:156, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.749686+0000", 2024-09-16T04:02:24.938 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.147188192000002, 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.021277265, 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749686+0000", 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749686+0000", 2024-09-16T04:02:24.939 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.940 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.940 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.940 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.940 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749688+0000", 2024-09-16T04:02:24.940 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.7060000000000001e-06 2024-09-16T04:02:24.940 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.940 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.940 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749738+0000", 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0303000000000003e-05 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749777+0000", 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.8399000000000003e-05 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.749800+0000", 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.3416000000000001e-05 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.941 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755725+0000", 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0059249259999999996 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755752+0000", 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.7219000000000001e-05 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.942 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770896+0000", 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015144316999999999 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770963+0000", 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.6978999999999997e-05 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.8s2 81/33 ECSubWrite(tid=28, reqid=osd.5.0:28, at_version=81'23, trim_to=0'0, roll_forward_to=79'21))", 2024-09-16T04:02:24.943 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.751718+0000", 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.145156230000001, 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019320789000000001, 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751718+0000", 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.944 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751718+0000", 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751719+0000", 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.4600000000000004e-07 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751791+0000", 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.2033000000000006e-05 2024-09-16T04:02:24.945 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.947 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.947 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.947 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751824+0000", 2024-09-16T04:02:24.947 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.3129000000000001e-05 2024-09-16T04:02:24.947 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.947 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.947 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.947 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.751846+0000", 2024-09-16T04:02:24.948 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.262e-05 2024-09-16T04:02:24.948 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.948 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.948 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755683+0000", 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0038367679999999999 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755706+0000", 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2829000000000001e-05 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770982+0000", 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015275739999999999 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.949 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771039+0000", 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7024000000000002e-05 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.fs1 81/33 ECSubWrite(tid=336, reqid=osd.11.0:336, at_version=81'70, trim_to=0'0, roll_forward_to=77'66))", 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.753557+0000", 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.143317491000001, 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.017652517999999999, 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.950 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753557+0000", 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753557+0000", 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.951 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753559+0000", 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8500000000000001e-06 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753636+0000", 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.7095000000000006e-05 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753666+0000", 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0406e-05 2024-09-16T04:02:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.753688+0000", 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1416e-05 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755220+0000", 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.001531988 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.953 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.755245+0000", 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.4983000000000001e-05 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771138+0000", 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015893442000000001 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.771209+0000", 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.1338e-05 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.954 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s5 81/33 ECSubWrite(tid=222, reqid=osd.4.0:222, at_version=81'33, trim_to=0'0, roll_forward_to=79'31))", 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.763721+0000", 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.133153134000001, 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013019978, 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763721+0000", 2024-09-16T04:02:24.955 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763721+0000", 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763723+0000", 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.9099999999999999e-06 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.956 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763799+0000", 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6068000000000004e-05 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763840+0000", 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.0720000000000003e-05 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.763862+0000", 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2538999999999999e-05 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.957 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.764054+0000", 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019151300000000001 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.764084+0000", 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.0583000000000002e-05 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.776658+0000", 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.012573457999999999 2024-09-16T04:02:24.958 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.776741+0000", 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.3187000000000002e-05 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s5 81/33 ECSubWrite(tid=224, reqid=osd.4.0:224, at_version=81'35, trim_to=0'0, roll_forward_to=79'31))", 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:06.766937+0000", 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 18.129937744999999, 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013032854, 2024-09-16T04:02:24.959 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.766937+0000", 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.766937+0000", 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.960 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.766938+0000", 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.886e-06 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.766999+0000", 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.0232999999999997e-05 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.767044+0000", 2024-09-16T04:02:24.961 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.5785000000000003e-05 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.767067+0000", 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.2744000000000001e-05 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770282+0000", 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0032146200000000001 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.962 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.770308+0000", 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6171999999999999e-05 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.779896+0000", 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0095884730000000001 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:06.779969+0000", 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.2941000000000005e-05 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.963 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.3s4 82/33 ECSubWrite(tid=285, reqid=osd.13.0:285, at_version=82'63, trim_to=0'0, roll_forward_to=81'61))", 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.774113+0000", 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 17.122761701000002, 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014629262000000001, 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774113+0000", 2024-09-16T04:02:24.964 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774113+0000", 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774113+0000", 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.9699999999999996e-07 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.965 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774169+0000", 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.5522000000000001e-05 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774194+0000", 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5032999999999999e-05 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774215+0000", 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1650000000000001e-05 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.966 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.967 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.967 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774423+0000", 2024-09-16T04:02:24.967 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000207395 2024-09-16T04:02:24.967 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.967 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.967 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.967 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.774448+0000", 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5259999999999999e-05 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788647+0000", 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014198489999999999 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788742+0000", 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.5315000000000005e-05 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s5 82/33 ECSubWrite(tid=233, reqid=osd.4.0:233, at_version=82'39, trim_to=0'0, roll_forward_to=81'35))", 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:07.775012+0000", 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 17.121862145000001, 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013457884999999999, 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.969 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775012+0000", 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775012+0000", 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775014+0000", 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.6559999999999999e-06 2024-09-16T04:02:24.970 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775067+0000", 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.2862999999999999e-05 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775098+0000", 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1272000000000001e-05 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775119+0000", 2024-09-16T04:02:24.971 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1290000000000001e-05 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775326+0000", 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00020677900000000001 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.775351+0000", 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5219e-05 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.972 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788373+0000", 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.013021721 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:07.788470+0000", 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.7084999999999996e-05 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.1s5 86/33 ECSubWrite(tid=265, reqid=osd.4.0:265, at_version=86'49, trim_to=0'0, roll_forward_to=85'47))", 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:11.742821+0000", 2024-09-16T04:02:24.973 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 13.154053247, 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.016916190000000001, 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.742821+0000", 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.974 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.742821+0000", 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.742823+0000", 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.5430000000000001e-06 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.742880+0000", 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.6988999999999999e-05 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.975 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.742912+0000", 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.2273000000000001e-05 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.742937+0000", 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.5380000000000001e-05 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.743114+0000", 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00017641999999999999 2024-09-16T04:02:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.743145+0000", 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.1183000000000003e-05 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.759635+0000", 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.016490337000000001 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.977 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.759737+0000", 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010206500000000001 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "MOSDECSubOpWrite(3.2s1 86/33 ECSubWrite(tid=298, reqid=osd.13.0:298, at_version=86'46, trim_to=0'0, roll_forward_to=84'44))", 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:11.745010+0000", 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 13.151864732, 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.014334606, 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "reached pg", 2024-09-16T04:02:24.978 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.745010+0000", 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.745010+0000", 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.745010+0000", 2024-09-16T04:02:24.979 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.7999999999999995e-07 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.745067+0000", 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.7099000000000003e-05 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.745094+0000", 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6582000000000001e-05 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.980 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.745116+0000", 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.1607000000000002e-05 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.745316+0000", 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00020019600000000001 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.745342+0000", 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.6267000000000001e-05 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.981 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.759242+0000", 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.01389964 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:11.759344+0000", 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00010253499999999999 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:951 3.es0 3:731b4600:::smithi18831615-23:head [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-16T04:02:24.982 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:17.738865+0000", 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 7.1580088960000001, 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.015094555000000001, 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 951 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.983 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738865+0000", 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738865+0000", 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738866+0000", 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 9.2299999999999999e-07 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.984 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738871+0000", 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 5.0139999999999998e-06 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738882+0000", 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.0643e-05 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738890+0000", 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 7.6140000000000003e-06 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.985 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.986 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.986 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.743516+0000", 2024-09-16T04:02:24.986 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0046268990000000003 2024-09-16T04:02:24.986 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.986 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.986 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:24.986 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.743837+0000", 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00032090500000000002 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.753960+0000", 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.010122557000000001 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:952 3.es0 3:731b4600:::smithi18831615-23:head [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:17.738901+0000", 2024-09-16T04:02:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 7.1579736780000003, 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.016524644000000002, 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "started", 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 952 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738901+0000", 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.988 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738901+0000", 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738901+0000", 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.1600000000000001e-07 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738906+0000", 2024-09-16T04:02:24.989 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 4.8640000000000004e-06 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738914+0000", 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 8.1489999999999994e-06 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.738920+0000", 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 6.122e-06 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.990 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.746350+0000", 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0074294069999999999 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.746512+0000", 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000161841 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:17.755425+0000", 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0089136449999999996 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:24.991 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "description": "osd_op(client.4646.0:963 3.es0 3:74948e3e:::smithi18831615-25:head [delete] snapc 28=[28,27,26,25,24,23,21,20,1f,1d,1b,19,18,17,16,15,11,f,e,d,a] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e93)", 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "initiated_at": "2024-09-16T04:02:18.768025+0000", 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "age": 6.1288491130000002, 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.019064940999999998, 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "continuous": false, 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "type_data": { 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "client_info": { 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "client": "client.4646", 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "client_addr": "172.21.15.188:0/3162109001", 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: "tid": 963 2024-09-16T04:02:24.992 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: "events": [ 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "initiated", 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.768025+0000", 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "header_read", 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.768025+0000", 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.993 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "throttled", 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.768028+0000", 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 2.9859999999999999e-06 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "all_read", 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.768042+0000", 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.3937000000000001e-05 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.996 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "dispatched", 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.768079+0000", 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 3.7036000000000003e-05 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "queued_for_pg", 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.768098+0000", 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.8533e-05 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.768269+0000", 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.000171077 2024-09-16T04:02:24.997 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "op must wait for map", 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.768283+0000", 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 1.385e-05 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "reached_pg", 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.773995+0000", 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0057124599999999999 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "started", 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.774271+0000", 2024-09-16T04:02:24.998 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00027606899999999999 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_started", 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.776111+0000", 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0018396109999999999 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "sub_op_committed", 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.779076+0000", 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0029655509999999999 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "commit_sent", 2024-09-16T04:02:24.999 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.786896+0000", 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.0078196099999999994 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: "event": "done", 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: "time": "2024-09-16T04:02:18.787090+0000", 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: "duration": 0.00019422100000000001 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-09-16T04:02:25.000 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-16T04:02:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-09-16T04:02:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:02:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:25.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:02:25.003 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:25.005 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:25.005 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 'trim stale osdmaps' 2024-09-16T04:02:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-09-16T04:02:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:02:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-09-16T04:02:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:02:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-09-16T04:02:25.012 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:02:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-09-16T04:02:25.029 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:02:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-09-16T04:02:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:02:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-09-16T04:02:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:02:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-09-16T04:02:25.087 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:02:25.118 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-09-16T04:02:25.118 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:02:25.135 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:25.135 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:25.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:02:25.150 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:25.150 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:25.150 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:25.150 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:25.150 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:25.150 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:25.150 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:25.150 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:25.151 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:25.151 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:25.151 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:25.151 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:25.152 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:25.152 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:25.153 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 'trim stale osdmaps' 2024-09-16T04:02:25.153 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-09-16T04:02:25.153 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:02:25.184 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-09-16T04:02:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:02:25.219 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-09-16T04:02:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:02:25.241 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-09-16T04:02:25.241 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:02:25.261 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-09-16T04:02:25.262 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:02:25.273 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:25.273 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:25.281 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-09-16T04:02:25.281 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:02:25.287 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:02:25.302 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-09-16T04:02:25.302 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:02:25.328 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:25.332 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:25.332 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 'trim stale osdmaps' 2024-09-16T04:02:25.410 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:25.410 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:25.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:02:25.522 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:25.522 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:25.523 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:25.525 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:25.526 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 'trim stale osdmaps' 2024-09-16T04:02:25.546 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:25.546 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:25.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:02:25.671 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2024-09-16T04:02:25.671 INFO:teuthology.orchestra.run.smithi037.stderr:0 2024-09-16T04:02:25.671 INFO:teuthology.orchestra.run.smithi037.stderr:1 2024-09-16T04:02:25.671 INFO:teuthology.orchestra.run.smithi037.stderr:2 2024-09-16T04:02:25.671 INFO:teuthology.orchestra.run.smithi037.stderr:abort 2024-09-16T04:02:25.671 INFO:teuthology.orchestra.run.smithi037.stderr:assert 2024-09-16T04:02:25.672 INFO:teuthology.orchestra.run.smithi037.stderr:bench [] [] [] [] 2024-09-16T04:02:25.672 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:02:25.672 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs files list 2024-09-16T04:02:25.672 INFO:teuthology.orchestra.run.smithi037.stderr:bluefs stats 2024-09-16T04:02:25.672 INFO:teuthology.orchestra.run.smithi037.stderr:bluestore allocator dump block 2024-09-16T04:02:25.672 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2024-09-16T04:02:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:02:25.674 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T04:02:25.681 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:25.681 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:25.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:02:25.812 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-09-16T04:02:25.821 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:25.821 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:25.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:02:25.956 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:25.957 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:25.970 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:02:26.087 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:26.088 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:26.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:02:26.230 INFO:teuthology.orchestra.run.smithi117.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:26.230 INFO:teuthology.orchestra.run.smithi117.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:26.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:02:26.377 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:26.377 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:26.391 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:02:26.522 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:26.522 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:26.536 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:02:26.658 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:26.658 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:26.672 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:02:26.789 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:26.789 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:26.805 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:02:26.926 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:26.926 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:26.940 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:02:27.061 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:27.061 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:27.075 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:02:27.192 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:27.193 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:27.207 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:02:27.322 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:27.322 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:27.336 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:02:27.460 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:27.460 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:27.474 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:02:27.591 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:27.591 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:27.606 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:02:27.730 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:27.730 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:27.744 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:02:27.835 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:02:27.860 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-09-16T04:02:27.860 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:02:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-09-16T04:02:28.851 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:02:29.845 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-09-16T04:02:30.675 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T04:02:30.828 INFO:teuthology.orchestra.run.smithi037.stdout:mon.a: {} 2024-09-16T04:02:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:02:30.842 INFO:teuthology.orchestra.run.smithi037.stdout:mon.b: {} 2024-09-16T04:02:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:02:30.856 INFO:teuthology.orchestra.run.smithi037.stdout:mon.c: {} 2024-09-16T04:02:30.856 INFO:teuthology.orchestra.run.smithi037.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:02:30.873 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T04:02:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T04:02:31.883 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T04:02:32.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T04:02:32.219+0000 7f33a5ea0640 0 --2- 172.21.15.37:0/3204087013 >> v2:172.21.15.117:3300/0 conn(0x7f33a014e890 0x7f33a00c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:02:32.859 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T04:02:32.885 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T04:02:33.874 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T04:02:33.901 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T04:02:34.877 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T04:02:34.904 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T04:02:35.893 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T04:02:35.920 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-16T04:02:36.895 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-16T04:02:36.922 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T04:02:37.912 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T04:02:37.935 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-16T04:02:38.927 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-16T04:02:38.949 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T04:02:39.936 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T04:02:39.960 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T04:02:40.938 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T04:02:40.965 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T04:02:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T04:02:41.976 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T04:02:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T04:02:42.990 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T04:02:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T04:02:44.004 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-16T04:02:44.984 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-16T04:02:45.008 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T04:02:46.001 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T04:02:46.025 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-16T04:02:47.004 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-16T04:02:47.034 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T04:02:48.023 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T04:02:48.047 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T04:02:49.026 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T04:02:49.053 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T04:02:50.043 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T04:02:50.068 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T04:02:51.053 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T04:02:51.079 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T04:02:52.063 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T04:02:52.091 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-16T04:02:53.066 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-16T04:02:53.094 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T04:02:54.083 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T04:02:54.110 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-16T04:02:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-16T04:02:55.112 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T04:02:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T04:02:56.122 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T04:02:57.118 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T04:02:57.148 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T04:02:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T04:02:58.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T04:02:59.133 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T04:02:59.161 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T04:03:00.147 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T04:03:00.171 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-16T04:03:01.156 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-16T04:03:01.183 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T04:03:02.169 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T04:03:02.196 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-16T04:03:03.182 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-16T04:03:03.211 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T04:03:03.211 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T04:03:03.211 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T04:03:03.212 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T04:03:03.212 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:03:03.212 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:03:03.524 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:03.525 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":133,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:03:03.178430+0000","last_up_change":"2024-09-16T04:00:57.414587+0000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:00:57.133339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:01:05.007470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6817","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6819","nonce":3422299298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6824","nonce":3422299298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6831","nonce":3422299298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6829","nonce":3422299298}]},"public_addr":"172.21.15.37:6819/3422299298","cluster_addr":"172.21.15.37:6824/3422299298","heartbeat_back_addr":"172.21.15.37:6831/3422299298","heartbeat_front_addr":"172.21.15.37:6829/3422299298","state":["exists","up"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6809","nonce":1856047451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6811","nonce":1856047451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6815","nonce":1856047451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6813","nonce":1856047451}]},"public_addr":"172.21.15.39:6809/1856047451","cluster_addr":"172.21.15.39:6811/1856047451","heartbeat_back_addr":"172.21.15.39:6815/1856047451","heartbeat_front_addr":"172.21.15.39:6813/1856047451","state":["exists","up"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6817","nonce":3343342478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6819","nonce":3343342478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6823","nonce":3343342478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6821","nonce":3343342478}]},"public_addr":"172.21.15.117:6817/3343342478","cluster_addr":"172.21.15.117:6819/3343342478","heartbeat_back_addr":"172.21.15.117:6823/3343342478","heartbeat_front_addr":"172.21.15.117:6821/3343342478","state":["exists","up"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6803","nonce":2905616722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6807","nonce":2905616722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6816","nonce":2905616722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6811","nonce":2905616722}]},"public_addr":"172.21.15.188:6803/2905616722","cluster_addr":"172.21.15.188:6807/2905616722","heartbeat_back_addr":"172.21.15.188:6816/2905616722","heartbeat_front_addr":"172.21.15.188:6811/2905616722","state":["exists","up"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6801","nonce":2698627343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6803","nonce":2698627343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6807","nonce":2698627343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6805","nonce":2698627343}]},"public_addr":"172.21.15.37:6801/2698627343","cluster_addr":"172.21.15.37:6803/2698627343","heartbeat_back_addr":"172.21.15.37:6807/2698627343","heartbeat_front_addr":"172.21.15.37:6805/2698627343","state":["exists","up"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6825","nonce":2772028538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6827","nonce":2772028538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6831","nonce":2772028538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6829","nonce":2772028538}]},"public_addr":"172.21.15.39:6825/2772028538","cluster_addr":"172.21.15.39:6827/2772028538","heartbeat_back_addr":"172.21.15.39:6831/2772028538","heartbeat_front_addr":"172.21.15.39:6829/2772028538","state":["exists","up"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6801","nonce":3124837418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6803","nonce":3124837418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6807","nonce":3124837418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6805","nonce":3124837418}]},"public_addr":"172.21.15.117:6801/3124837418","cluster_addr":"172.21.15.117:6803/3124837418","heartbeat_back_addr":"172.21.15.117:6807/3124837418","heartbeat_front_addr":"172.21.15.117:6805/3124837418","state":["exists","up"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6806","nonce":4221150049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6810","nonce":4221150049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6815","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6817","nonce":4221150049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6813","nonce":4221150049}]},"public_addr":"172.21.15.188:6806/4221150049","cluster_addr":"172.21.15.188:6810/4221150049","heartbeat_back_addr":"172.21.15.188:6817/4221150049","heartbeat_front_addr":"172.21.15.188:6813/4221150049","state":["exists","up"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6809","nonce":3581510146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6811","nonce":3581510146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6815","nonce":3581510146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6813","nonce":3581510146}]},"public_addr":"172.21.15.37:6809/3581510146","cluster_addr":"172.21.15.37:6811/3581510146","heartbeat_back_addr":"172.21.15.37:6815/3581510146","heartbeat_front_addr":"172.21.15.37:6813/3581510146","state":["exists","up"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6801","nonce":2726119652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6803","nonce":2726119652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6807","nonce":2726119652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6805","nonce":2726119652}]},"public_addr":"172.21.15.39:6801/2726119652","cluster_addr":"172.21.15.39:6803/2726119652","heartbeat_back_addr":"172.21.15.39:6807/2726119652","heartbeat_front_addr":"172.21.15.39:6805/2726119652","state":["exists","up"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6825","nonce":737103121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6827","nonce":737103121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6831","nonce":737103121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6829","nonce":737103121}]},"public_addr":"172.21.15.117:6825/737103121","cluster_addr":"172.21.15.117:6827/737103121","heartbeat_back_addr":"172.21.15.117:6831/737103121","heartbeat_front_addr":"172.21.15.117:6829/737103121","state":["exists","up"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6823","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6824","nonce":1355328456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6828","nonce":1355328456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6833","nonce":1355328456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6831","nonce":1355328456}]},"public_addr":"172.21.15.188:6824/1355328456","cluster_addr":"172.21.15.188:6828/1355328456","heartbeat_back_addr":"172.21.15.188:6833/1355328456","heartbeat_front_addr":"172.21.15.188:6831/1355328456","state":["exists","up"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6818","nonce":3693386273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6821","nonce":3693386273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6827","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6828","nonce":3693386273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6825","nonce":3693386273}]},"public_addr":"172.21.15.37:6818/3693386273","cluster_addr":"172.21.15.37:6821/3693386273","heartbeat_back_addr":"172.21.15.37:6828/3693386273","heartbeat_front_addr":"172.21.15.37:6825/3693386273","state":["exists","up"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6817","nonce":2905392027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6819","nonce":2905392027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6823","nonce":2905392027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6821","nonce":2905392027}]},"public_addr":"172.21.15.39:6817/2905392027","cluster_addr":"172.21.15.39:6819/2905392027","heartbeat_back_addr":"172.21.15.39:6823/2905392027","heartbeat_front_addr":"172.21.15.39:6821/2905392027","state":["exists","up"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6809","nonce":1375514506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6811","nonce":1375514506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6815","nonce":1375514506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6813","nonce":1375514506}]},"public_addr":"172.21.15.117:6809/1375514506","cluster_addr":"172.21.15.117:6811/1375514506","heartbeat_back_addr":"172.21.15.117:6815/1375514506","heartbeat_front_addr":"172.21.15.117:6813/1375514506","state":["exists","up"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6819","nonce":3000701187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6821","nonce":3000701187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6829","nonce":3000701187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6825","nonce":3000701187}]},"public_addr":"172.21.15.188:6819/3000701187","cluster_addr":"172.21.15.188:6821/3000701187","heartbeat_back_addr":"172.21.15.188:6829/3000701187","heartbeat_front_addr":"172.21.15.188:6825/3000701187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.245592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:05.860319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:06.602619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:58.884947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:15.931876+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:54.754303+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:56.478181+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:01.375477+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:57.934680+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:17.873726+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:57.781946+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:16.811116+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.982880+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.914975+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:08.616813+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:06.605685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":17,"length":1},{"begin":30,"length":1},{"begin":35,"length":1},{"begin":38,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:03:03.544 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:03:03.545 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:03:03.851 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:03.851 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":133,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:03:03.178430+0000","last_up_change":"2024-09-16T04:00:57.414587+0000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:00:57.133339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:01:05.007470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6817","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6819","nonce":3422299298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6824","nonce":3422299298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6831","nonce":3422299298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6829","nonce":3422299298}]},"public_addr":"172.21.15.37:6819/3422299298","cluster_addr":"172.21.15.37:6824/3422299298","heartbeat_back_addr":"172.21.15.37:6831/3422299298","heartbeat_front_addr":"172.21.15.37:6829/3422299298","state":["exists","up"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6809","nonce":1856047451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6811","nonce":1856047451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6815","nonce":1856047451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6813","nonce":1856047451}]},"public_addr":"172.21.15.39:6809/1856047451","cluster_addr":"172.21.15.39:6811/1856047451","heartbeat_back_addr":"172.21.15.39:6815/1856047451","heartbeat_front_addr":"172.21.15.39:6813/1856047451","state":["exists","up"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6817","nonce":3343342478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6819","nonce":3343342478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6823","nonce":3343342478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6821","nonce":3343342478}]},"public_addr":"172.21.15.117:6817/3343342478","cluster_addr":"172.21.15.117:6819/3343342478","heartbeat_back_addr":"172.21.15.117:6823/3343342478","heartbeat_front_addr":"172.21.15.117:6821/3343342478","state":["exists","up"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6803","nonce":2905616722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6807","nonce":2905616722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6816","nonce":2905616722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6811","nonce":2905616722}]},"public_addr":"172.21.15.188:6803/2905616722","cluster_addr":"172.21.15.188:6807/2905616722","heartbeat_back_addr":"172.21.15.188:6816/2905616722","heartbeat_front_addr":"172.21.15.188:6811/2905616722","state":["exists","up"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6801","nonce":2698627343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6803","nonce":2698627343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6807","nonce":2698627343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6805","nonce":2698627343}]},"public_addr":"172.21.15.37:6801/2698627343","cluster_addr":"172.21.15.37:6803/2698627343","heartbeat_back_addr":"172.21.15.37:6807/2698627343","heartbeat_front_addr":"172.21.15.37:6805/2698627343","state":["exists","up"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6825","nonce":2772028538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6827","nonce":2772028538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6831","nonce":2772028538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6829","nonce":2772028538}]},"public_addr":"172.21.15.39:6825/2772028538","cluster_addr":"172.21.15.39:6827/2772028538","heartbeat_back_addr":"172.21.15.39:6831/2772028538","heartbeat_front_addr":"172.21.15.39:6829/2772028538","state":["exists","up"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6801","nonce":3124837418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6803","nonce":3124837418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6807","nonce":3124837418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6805","nonce":3124837418}]},"public_addr":"172.21.15.117:6801/3124837418","cluster_addr":"172.21.15.117:6803/3124837418","heartbeat_back_addr":"172.21.15.117:6807/3124837418","heartbeat_front_addr":"172.21.15.117:6805/3124837418","state":["exists","up"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6806","nonce":4221150049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6810","nonce":4221150049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6815","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6817","nonce":4221150049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6813","nonce":4221150049}]},"public_addr":"172.21.15.188:6806/4221150049","cluster_addr":"172.21.15.188:6810/4221150049","heartbeat_back_addr":"172.21.15.188:6817/4221150049","heartbeat_front_addr":"172.21.15.188:6813/4221150049","state":["exists","up"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6809","nonce":3581510146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6811","nonce":3581510146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6815","nonce":3581510146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6813","nonce":3581510146}]},"public_addr":"172.21.15.37:6809/3581510146","cluster_addr":"172.21.15.37:6811/3581510146","heartbeat_back_addr":"172.21.15.37:6815/3581510146","heartbeat_front_addr":"172.21.15.37:6813/3581510146","state":["exists","up"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6801","nonce":2726119652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6803","nonce":2726119652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6807","nonce":2726119652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6805","nonce":2726119652}]},"public_addr":"172.21.15.39:6801/2726119652","cluster_addr":"172.21.15.39:6803/2726119652","heartbeat_back_addr":"172.21.15.39:6807/2726119652","heartbeat_front_addr":"172.21.15.39:6805/2726119652","state":["exists","up"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6825","nonce":737103121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6827","nonce":737103121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6831","nonce":737103121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6829","nonce":737103121}]},"public_addr":"172.21.15.117:6825/737103121","cluster_addr":"172.21.15.117:6827/737103121","heartbeat_back_addr":"172.21.15.117:6831/737103121","heartbeat_front_addr":"172.21.15.117:6829/737103121","state":["exists","up"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6823","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6824","nonce":1355328456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6828","nonce":1355328456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6833","nonce":1355328456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6831","nonce":1355328456}]},"public_addr":"172.21.15.188:6824/1355328456","cluster_addr":"172.21.15.188:6828/1355328456","heartbeat_back_addr":"172.21.15.188:6833/1355328456","heartbeat_front_addr":"172.21.15.188:6831/1355328456","state":["exists","up"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6818","nonce":3693386273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6821","nonce":3693386273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6827","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6828","nonce":3693386273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6825","nonce":3693386273}]},"public_addr":"172.21.15.37:6818/3693386273","cluster_addr":"172.21.15.37:6821/3693386273","heartbeat_back_addr":"172.21.15.37:6828/3693386273","heartbeat_front_addr":"172.21.15.37:6825/3693386273","state":["exists","up"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6817","nonce":2905392027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6819","nonce":2905392027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6823","nonce":2905392027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6821","nonce":2905392027}]},"public_addr":"172.21.15.39:6817/2905392027","cluster_addr":"172.21.15.39:6819/2905392027","heartbeat_back_addr":"172.21.15.39:6823/2905392027","heartbeat_front_addr":"172.21.15.39:6821/2905392027","state":["exists","up"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6809","nonce":1375514506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6811","nonce":1375514506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6815","nonce":1375514506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6813","nonce":1375514506}]},"public_addr":"172.21.15.117:6809/1375514506","cluster_addr":"172.21.15.117:6811/1375514506","heartbeat_back_addr":"172.21.15.117:6815/1375514506","heartbeat_front_addr":"172.21.15.117:6813/1375514506","state":["exists","up"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6819","nonce":3000701187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6821","nonce":3000701187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6829","nonce":3000701187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6825","nonce":3000701187}]},"public_addr":"172.21.15.188:6819/3000701187","cluster_addr":"172.21.15.188:6821/3000701187","heartbeat_back_addr":"172.21.15.188:6829/3000701187","heartbeat_front_addr":"172.21.15.188:6825/3000701187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.245592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:05.860319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:06.602619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:58.884947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:15.931876+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:54.754303+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:56.478181+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:01.375477+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:57.934680+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:17.873726+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:57.781946+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:16.811116+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.982880+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.914975+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:08.616813+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:06.605685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":17,"length":1},{"begin":30,"length":1},{"begin":35,"length":1},{"begin":38,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:03:03.871 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:03:03.871 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:03:03.871 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:03:03.872 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:03:03.872 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T04:03:03.873 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T04:03:03.873 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T04:03:03.873 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T04:03:03.874 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T04:03:03.874 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T04:03:03.875 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T04:03:03.875 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T04:03:03.876 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T04:03:03.876 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T04:03:03.877 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T04:03:03.877 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T04:03:04.314 INFO:teuthology.orchestra.run.smithi037.stdout:98784247837 2024-09-16T04:03:04.315 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:03:04.342 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-09-16T04:03:04.342 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T04:03:04.343 INFO:teuthology.orchestra.run.smithi037.stdout:94489280542 2024-09-16T04:03:04.343 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:03:04.345 INFO:teuthology.orchestra.run.smithi037.stdout:103079215135 2024-09-16T04:03:04.345 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:03:04.361 INFO:teuthology.orchestra.run.smithi037.stdout:103079215134 2024-09-16T04:03:04.361 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:03:04.363 INFO:teuthology.orchestra.run.smithi037.stdout:94489280543 2024-09-16T04:03:04.363 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:03:04.385 INFO:teuthology.orchestra.run.smithi037.stdout:103079215134 2024-09-16T04:03:04.386 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:03:04.402 INFO:teuthology.orchestra.run.smithi037.stdout:98784247838 2024-09-16T04:03:04.403 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:03:04.427 INFO:teuthology.orchestra.run.smithi037.stdout:103079215134 2024-09-16T04:03:04.428 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T04:03:04.428 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-09-16T04:03:04.429 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:03:04.437 INFO:teuthology.orchestra.run.smithi037.stdout:94489280542 2024-09-16T04:03:04.437 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T04:03:04.446 INFO:teuthology.orchestra.run.smithi037.stdout:98784247836 2024-09-16T04:03:04.446 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:03:04.454 INFO:teuthology.orchestra.run.smithi037.stdout:98784247838 2024-09-16T04:03:04.454 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T04:03:04.455 INFO:teuthology.orchestra.run.smithi037.stdout:98784247838 2024-09-16T04:03:04.455 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:03:04.458 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-09-16T04:03:04.459 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:03:04.468 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-09-16T04:03:04.468 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:03:05.288 INFO:teuthology.orchestra.run.smithi037.stdout:98784247837 2024-09-16T04:03:05.310 INFO:teuthology.orchestra.run.smithi037.stdout:94489280542 2024-09-16T04:03:05.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247837 got 98784247837 for osd.0 2024-09-16T04:03:05.335 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.352 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-09-16T04:03:05.379 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.1 2024-09-16T04:03:05.379 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.401 INFO:teuthology.orchestra.run.smithi037.stdout:103079215135 2024-09-16T04:03:05.402 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.11 2024-09-16T04:03:05.402 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.434 INFO:teuthology.orchestra.run.smithi037.stdout:103079215134 2024-09-16T04:03:05.444 INFO:teuthology.orchestra.run.smithi037.stdout:98784247838 2024-09-16T04:03:05.452 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215135 got 103079215135 for osd.6 2024-09-16T04:03:05.452 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.486 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215134 got 103079215134 for osd.2 2024-09-16T04:03:05.486 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.487 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-09-16T04:03:05.491 INFO:teuthology.orchestra.run.smithi037.stdout:94489280543 2024-09-16T04:03:05.491 INFO:teuthology.orchestra.run.smithi037.stdout:98784247838 2024-09-16T04:03:05.495 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247838 got 98784247838 for osd.5 2024-09-16T04:03:05.496 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.513 INFO:teuthology.orchestra.run.smithi037.stdout:98784247836 2024-09-16T04:03:05.525 INFO:teuthology.orchestra.run.smithi037.stdout:103079215134 2024-09-16T04:03:05.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280543 got 94489280543 for osd.4 2024-09-16T04:03:05.525 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.527 INFO:teuthology.orchestra.run.smithi037.stdout:103079215134 2024-09-16T04:03:05.527 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-09-16T04:03:05.529 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-09-16T04:03:05.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.12 2024-09-16T04:03:05.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247838 got 98784247838 for osd.13 2024-09-16T04:03:05.533 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.533 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.535 INFO:teuthology.orchestra.run.smithi037.stdout:94489280542 2024-09-16T04:03:05.549 INFO:teuthology.orchestra.run.smithi037.stdout:98784247838 2024-09-16T04:03:05.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247836 got 98784247836 for osd.7 2024-09-16T04:03:05.559 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.565 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215134 got 103079215134 for osd.10 2024-09-16T04:03:05.565 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.9 2024-09-16T04:03:05.568 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.569 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215134 got 103079215134 for osd.14 2024-09-16T04:03:05.569 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.8 2024-09-16T04:03:05.570 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.571 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.15 2024-09-16T04:03:05.572 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247838 got 98784247838 for osd.3 2024-09-16T04:03:05.575 DEBUG:teuthology.parallel:result is None 2024-09-16T04:03:05.576 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T04:03:05.576 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:03:05.894 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:03:05.910 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T04:03:05.076163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":635524,"kb_used_data":32344,"kb_used_omap":24,"kb_used_meta":602791,"kb_avail":1509313916,"statfs":{"total":1546188226560,"available":1545537449984,"internally_reserved":0,"allocated":33120256,"data_stored":19730214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":617258065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.928718"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":425,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:16.104902+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T04:02:16.104676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:02:16.104676+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:30.120742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.558640+0000","last_change":"2024-09-16T04:02:13.919586+0000","last_active":"2024-09-16T04:03:03.558640+0000","last_peered":"2024-09-16T04:03:03.558640+0000","last_clean":"2024-09-16T04:03:03.558640+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:03:03.558640+0000","last_undegraded":"2024-09-16T04:03:03.558640+0000","last_fullsized":"2024-09-16T04:03:03.558640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'1","last_scrub_stamp":"2024-09-16T04:02:13.919386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:13.919386+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:39.323146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185421+0000","last_change":"2024-09-16T04:02:31.973520+0000","last_active":"2024-09-16T04:03:03.185421+0000","last_peered":"2024-09-16T04:03:03.185421+0000","last_clean":"2024-09-16T04:03:03.185421+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:03:03.185421+0000","last_undegraded":"2024-09-16T04:03:03.185421+0000","last_fullsized":"2024-09-16T04:03:03.185421+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.973151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.973151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:49.066570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":227,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:30.059627+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:30.059406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:30.059406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:57.455661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"105'3","reported_seq":225,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185459+0000","last_change":"2024-09-16T04:02:37.020433+0000","last_active":"2024-09-16T04:03:03.185459+0000","last_peered":"2024-09-16T04:03:03.185459+0000","last_clean":"2024-09-16T04:03:03.185459+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:03:03.185459+0000","last_undegraded":"2024-09-16T04:03:03.185459+0000","last_fullsized":"2024-09-16T04:03:03.185459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_clean_scrub_stamp":"2024-09-16T04:02:37.020087+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:45.933828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185254+0000","last_change":"2024-09-16T04:02:31.948942+0000","last_active":"2024-09-16T04:03:03.185254+0000","last_peered":"2024-09-16T04:03:03.185254+0000","last_clean":"2024-09-16T04:03:03.185254+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:03:03.185254+0000","last_undegraded":"2024-09-16T04:03:03.185254+0000","last_fullsized":"2024-09-16T04:03:03.185254+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.948810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:00.770248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.873784+0000","last_change":"2024-09-16T04:02:26.870604+0000","last_active":"2024-09-16T04:03:03.873784+0000","last_peered":"2024-09-16T04:03:03.873784+0000","last_clean":"2024-09-16T04:03:03.873784+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:03:03.873784+0000","last_undegraded":"2024-09-16T04:03:03.873784+0000","last_fullsized":"2024-09-16T04:03:03.873784+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.870379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.870379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:35.995441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.872222+0000","last_change":"2024-09-16T04:02:36.511128+0000","last_active":"2024-09-16T04:03:03.872222+0000","last_peered":"2024-09-16T04:03:03.872222+0000","last_clean":"2024-09-16T04:03:03.872222+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:03:03.872222+0000","last_undegraded":"2024-09-16T04:03:03.872222+0000","last_fullsized":"2024-09-16T04:03:03.872222+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_clean_scrub_stamp":"2024-09-16T04:02:36.510902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:55.939407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":224,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137419+0000","last_change":"2024-09-16T04:02:26.602127+0000","last_active":"2024-09-16T04:03:04.137419+0000","last_peered":"2024-09-16T04:03:04.137419+0000","last_clean":"2024-09-16T04:03:04.137419+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:03:04.137419+0000","last_undegraded":"2024-09-16T04:03:04.137419+0000","last_fullsized":"2024-09-16T04:03:04.137419+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.601970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.601970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:46.647581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51204,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320636,"statfs":{"total":96636764160,"available":96584331264,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215135,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51208,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320632,"statfs":{"total":96636764160,"available":96584327168,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51212,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320628,"statfs":{"total":96636764160,"available":96584323072,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,3,5,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28996,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342844,"statfs":{"total":96636764160,"available":96607072256,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33156,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28980,"kb_used_data":1952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342860,"statfs":{"total":96636764160,"available":96607088640,"internally_reserved":0,"allocated":1998848,"data_stored":1162222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28976,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342864,"statfs":{"total":96636764160,"available":96607092736,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51720,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320120,"statfs":{"total":96636764160,"available":96583802880,"internally_reserved":0,"allocated":2543616,"data_stored":1700765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51184,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320656,"statfs":{"total":96636764160,"available":96584351744,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2527232,"data_stored":1691131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33160,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338680,"statfs":{"total":96636764160,"available":96602808320,"internally_reserved":0,"allocated":2015232,"data_stored":1171856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55364,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316476,"statfs":{"total":96636764160,"available":96580071424,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247836,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:03:05.912 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:03:06.193 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:06.193 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:03:06.209 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T04:03:05.076163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":635524,"kb_used_data":32344,"kb_used_omap":24,"kb_used_meta":602791,"kb_avail":1509313916,"statfs":{"total":1546188226560,"available":1545537449984,"internally_reserved":0,"allocated":33120256,"data_stored":19730214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":617258065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.928718"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":425,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:16.104902+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T04:02:16.104676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:02:16.104676+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:30.120742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.558640+0000","last_change":"2024-09-16T04:02:13.919586+0000","last_active":"2024-09-16T04:03:03.558640+0000","last_peered":"2024-09-16T04:03:03.558640+0000","last_clean":"2024-09-16T04:03:03.558640+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:03:03.558640+0000","last_undegraded":"2024-09-16T04:03:03.558640+0000","last_fullsized":"2024-09-16T04:03:03.558640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'1","last_scrub_stamp":"2024-09-16T04:02:13.919386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:13.919386+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:39.323146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185421+0000","last_change":"2024-09-16T04:02:31.973520+0000","last_active":"2024-09-16T04:03:03.185421+0000","last_peered":"2024-09-16T04:03:03.185421+0000","last_clean":"2024-09-16T04:03:03.185421+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:03:03.185421+0000","last_undegraded":"2024-09-16T04:03:03.185421+0000","last_fullsized":"2024-09-16T04:03:03.185421+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.973151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.973151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:49.066570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":227,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:30.059627+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:30.059406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:30.059406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:57.455661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"105'3","reported_seq":225,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185459+0000","last_change":"2024-09-16T04:02:37.020433+0000","last_active":"2024-09-16T04:03:03.185459+0000","last_peered":"2024-09-16T04:03:03.185459+0000","last_clean":"2024-09-16T04:03:03.185459+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:03:03.185459+0000","last_undegraded":"2024-09-16T04:03:03.185459+0000","last_fullsized":"2024-09-16T04:03:03.185459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_clean_scrub_stamp":"2024-09-16T04:02:37.020087+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:45.933828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185254+0000","last_change":"2024-09-16T04:02:31.948942+0000","last_active":"2024-09-16T04:03:03.185254+0000","last_peered":"2024-09-16T04:03:03.185254+0000","last_clean":"2024-09-16T04:03:03.185254+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:03:03.185254+0000","last_undegraded":"2024-09-16T04:03:03.185254+0000","last_fullsized":"2024-09-16T04:03:03.185254+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.948810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:00.770248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.873784+0000","last_change":"2024-09-16T04:02:26.870604+0000","last_active":"2024-09-16T04:03:03.873784+0000","last_peered":"2024-09-16T04:03:03.873784+0000","last_clean":"2024-09-16T04:03:03.873784+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:03:03.873784+0000","last_undegraded":"2024-09-16T04:03:03.873784+0000","last_fullsized":"2024-09-16T04:03:03.873784+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.870379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.870379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:35.995441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.872222+0000","last_change":"2024-09-16T04:02:36.511128+0000","last_active":"2024-09-16T04:03:03.872222+0000","last_peered":"2024-09-16T04:03:03.872222+0000","last_clean":"2024-09-16T04:03:03.872222+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:03:03.872222+0000","last_undegraded":"2024-09-16T04:03:03.872222+0000","last_fullsized":"2024-09-16T04:03:03.872222+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_clean_scrub_stamp":"2024-09-16T04:02:36.510902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:55.939407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":224,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137419+0000","last_change":"2024-09-16T04:02:26.602127+0000","last_active":"2024-09-16T04:03:04.137419+0000","last_peered":"2024-09-16T04:03:04.137419+0000","last_clean":"2024-09-16T04:03:04.137419+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:03:04.137419+0000","last_undegraded":"2024-09-16T04:03:04.137419+0000","last_fullsized":"2024-09-16T04:03:04.137419+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.601970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.601970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:46.647581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51204,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320636,"statfs":{"total":96636764160,"available":96584331264,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215135,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51208,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320632,"statfs":{"total":96636764160,"available":96584327168,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51212,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320628,"statfs":{"total":96636764160,"available":96584323072,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,3,5,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28996,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342844,"statfs":{"total":96636764160,"available":96607072256,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33156,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28980,"kb_used_data":1952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342860,"statfs":{"total":96636764160,"available":96607088640,"internally_reserved":0,"allocated":1998848,"data_stored":1162222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28976,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342864,"statfs":{"total":96636764160,"available":96607092736,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51720,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320120,"statfs":{"total":96636764160,"available":96583802880,"internally_reserved":0,"allocated":2543616,"data_stored":1700765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51184,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320656,"statfs":{"total":96636764160,"available":96584351744,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2527232,"data_stored":1691131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33160,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338680,"statfs":{"total":96636764160,"available":96602808320,"internally_reserved":0,"allocated":2015232,"data_stored":1171856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55364,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316476,"statfs":{"total":96636764160,"available":96580071424,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247836,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:03:06.211 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T04:03:06.211 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T04:03:06.223 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:03:06.224 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:03:06.513 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:06.514 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:03:06.531 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T04:03:05.076163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":635524,"kb_used_data":32344,"kb_used_omap":24,"kb_used_meta":602791,"kb_avail":1509313916,"statfs":{"total":1546188226560,"available":1545537449984,"internally_reserved":0,"allocated":33120256,"data_stored":19730214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":617258065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.928718"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":425,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:16.104902+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T04:02:16.104676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:02:16.104676+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:30.120742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.558640+0000","last_change":"2024-09-16T04:02:13.919586+0000","last_active":"2024-09-16T04:03:03.558640+0000","last_peered":"2024-09-16T04:03:03.558640+0000","last_clean":"2024-09-16T04:03:03.558640+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:03:03.558640+0000","last_undegraded":"2024-09-16T04:03:03.558640+0000","last_fullsized":"2024-09-16T04:03:03.558640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'1","last_scrub_stamp":"2024-09-16T04:02:13.919386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:13.919386+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:39.323146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185421+0000","last_change":"2024-09-16T04:02:31.973520+0000","last_active":"2024-09-16T04:03:03.185421+0000","last_peered":"2024-09-16T04:03:03.185421+0000","last_clean":"2024-09-16T04:03:03.185421+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:03:03.185421+0000","last_undegraded":"2024-09-16T04:03:03.185421+0000","last_fullsized":"2024-09-16T04:03:03.185421+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.973151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.973151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:49.066570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":227,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:30.059627+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:30.059406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:30.059406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:57.455661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"105'3","reported_seq":225,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185459+0000","last_change":"2024-09-16T04:02:37.020433+0000","last_active":"2024-09-16T04:03:03.185459+0000","last_peered":"2024-09-16T04:03:03.185459+0000","last_clean":"2024-09-16T04:03:03.185459+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:03:03.185459+0000","last_undegraded":"2024-09-16T04:03:03.185459+0000","last_fullsized":"2024-09-16T04:03:03.185459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_clean_scrub_stamp":"2024-09-16T04:02:37.020087+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:45.933828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185254+0000","last_change":"2024-09-16T04:02:31.948942+0000","last_active":"2024-09-16T04:03:03.185254+0000","last_peered":"2024-09-16T04:03:03.185254+0000","last_clean":"2024-09-16T04:03:03.185254+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:03:03.185254+0000","last_undegraded":"2024-09-16T04:03:03.185254+0000","last_fullsized":"2024-09-16T04:03:03.185254+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.948810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:00.770248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.873784+0000","last_change":"2024-09-16T04:02:26.870604+0000","last_active":"2024-09-16T04:03:03.873784+0000","last_peered":"2024-09-16T04:03:03.873784+0000","last_clean":"2024-09-16T04:03:03.873784+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:03:03.873784+0000","last_undegraded":"2024-09-16T04:03:03.873784+0000","last_fullsized":"2024-09-16T04:03:03.873784+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.870379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.870379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:35.995441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.872222+0000","last_change":"2024-09-16T04:02:36.511128+0000","last_active":"2024-09-16T04:03:03.872222+0000","last_peered":"2024-09-16T04:03:03.872222+0000","last_clean":"2024-09-16T04:03:03.872222+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:03:03.872222+0000","last_undegraded":"2024-09-16T04:03:03.872222+0000","last_fullsized":"2024-09-16T04:03:03.872222+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_clean_scrub_stamp":"2024-09-16T04:02:36.510902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:55.939407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":224,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137419+0000","last_change":"2024-09-16T04:02:26.602127+0000","last_active":"2024-09-16T04:03:04.137419+0000","last_peered":"2024-09-16T04:03:04.137419+0000","last_clean":"2024-09-16T04:03:04.137419+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:03:04.137419+0000","last_undegraded":"2024-09-16T04:03:04.137419+0000","last_fullsized":"2024-09-16T04:03:04.137419+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.601970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.601970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:46.647581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51204,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320636,"statfs":{"total":96636764160,"available":96584331264,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215135,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51208,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320632,"statfs":{"total":96636764160,"available":96584327168,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51212,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320628,"statfs":{"total":96636764160,"available":96584323072,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,3,5,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28996,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342844,"statfs":{"total":96636764160,"available":96607072256,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33156,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28980,"kb_used_data":1952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342860,"statfs":{"total":96636764160,"available":96607088640,"internally_reserved":0,"allocated":1998848,"data_stored":1162222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28976,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342864,"statfs":{"total":96636764160,"available":96607092736,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51720,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320120,"statfs":{"total":96636764160,"available":96583802880,"internally_reserved":0,"allocated":2543616,"data_stored":1700765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51184,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320656,"statfs":{"total":96636764160,"available":96584351744,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2527232,"data_stored":1691131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33160,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338680,"statfs":{"total":96636764160,"available":96602808320,"internally_reserved":0,"allocated":2015232,"data_stored":1171856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55364,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316476,"statfs":{"total":96636764160,"available":96580071424,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247836,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:03:06.532 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:03:06.816 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:06.817 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:03:06.833 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T04:03:05.076163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":635524,"kb_used_data":32344,"kb_used_omap":24,"kb_used_meta":602791,"kb_avail":1509313916,"statfs":{"total":1546188226560,"available":1545537449984,"internally_reserved":0,"allocated":33120256,"data_stored":19730214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":617258065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.928718"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":425,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:16.104902+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T04:02:16.104676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:02:16.104676+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:30.120742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.558640+0000","last_change":"2024-09-16T04:02:13.919586+0000","last_active":"2024-09-16T04:03:03.558640+0000","last_peered":"2024-09-16T04:03:03.558640+0000","last_clean":"2024-09-16T04:03:03.558640+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:03:03.558640+0000","last_undegraded":"2024-09-16T04:03:03.558640+0000","last_fullsized":"2024-09-16T04:03:03.558640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'1","last_scrub_stamp":"2024-09-16T04:02:13.919386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:13.919386+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:39.323146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185421+0000","last_change":"2024-09-16T04:02:31.973520+0000","last_active":"2024-09-16T04:03:03.185421+0000","last_peered":"2024-09-16T04:03:03.185421+0000","last_clean":"2024-09-16T04:03:03.185421+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:03:03.185421+0000","last_undegraded":"2024-09-16T04:03:03.185421+0000","last_fullsized":"2024-09-16T04:03:03.185421+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.973151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.973151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:49.066570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":227,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:30.059627+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:30.059406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:30.059406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:57.455661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"105'3","reported_seq":225,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185459+0000","last_change":"2024-09-16T04:02:37.020433+0000","last_active":"2024-09-16T04:03:03.185459+0000","last_peered":"2024-09-16T04:03:03.185459+0000","last_clean":"2024-09-16T04:03:03.185459+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:03:03.185459+0000","last_undegraded":"2024-09-16T04:03:03.185459+0000","last_fullsized":"2024-09-16T04:03:03.185459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_clean_scrub_stamp":"2024-09-16T04:02:37.020087+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:45.933828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185254+0000","last_change":"2024-09-16T04:02:31.948942+0000","last_active":"2024-09-16T04:03:03.185254+0000","last_peered":"2024-09-16T04:03:03.185254+0000","last_clean":"2024-09-16T04:03:03.185254+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:03:03.185254+0000","last_undegraded":"2024-09-16T04:03:03.185254+0000","last_fullsized":"2024-09-16T04:03:03.185254+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.948810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:00.770248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.873784+0000","last_change":"2024-09-16T04:02:26.870604+0000","last_active":"2024-09-16T04:03:03.873784+0000","last_peered":"2024-09-16T04:03:03.873784+0000","last_clean":"2024-09-16T04:03:03.873784+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:03:03.873784+0000","last_undegraded":"2024-09-16T04:03:03.873784+0000","last_fullsized":"2024-09-16T04:03:03.873784+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.870379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.870379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:35.995441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.872222+0000","last_change":"2024-09-16T04:02:36.511128+0000","last_active":"2024-09-16T04:03:03.872222+0000","last_peered":"2024-09-16T04:03:03.872222+0000","last_clean":"2024-09-16T04:03:03.872222+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:03:03.872222+0000","last_undegraded":"2024-09-16T04:03:03.872222+0000","last_fullsized":"2024-09-16T04:03:03.872222+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_clean_scrub_stamp":"2024-09-16T04:02:36.510902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:55.939407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":224,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137419+0000","last_change":"2024-09-16T04:02:26.602127+0000","last_active":"2024-09-16T04:03:04.137419+0000","last_peered":"2024-09-16T04:03:04.137419+0000","last_clean":"2024-09-16T04:03:04.137419+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:03:04.137419+0000","last_undegraded":"2024-09-16T04:03:04.137419+0000","last_fullsized":"2024-09-16T04:03:04.137419+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.601970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.601970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:46.647581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51204,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320636,"statfs":{"total":96636764160,"available":96584331264,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215135,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51208,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320632,"statfs":{"total":96636764160,"available":96584327168,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51212,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320628,"statfs":{"total":96636764160,"available":96584323072,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,3,5,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28996,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342844,"statfs":{"total":96636764160,"available":96607072256,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33156,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28980,"kb_used_data":1952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342860,"statfs":{"total":96636764160,"available":96607088640,"internally_reserved":0,"allocated":1998848,"data_stored":1162222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28976,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342864,"statfs":{"total":96636764160,"available":96607092736,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51720,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320120,"statfs":{"total":96636764160,"available":96583802880,"internally_reserved":0,"allocated":2543616,"data_stored":1700765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51184,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320656,"statfs":{"total":96636764160,"available":96584351744,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2527232,"data_stored":1691131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33160,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338680,"statfs":{"total":96636764160,"available":96602808320,"internally_reserved":0,"allocated":2015232,"data_stored":1171856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55364,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316476,"statfs":{"total":96636764160,"available":96580071424,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247836,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:03:06.834 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:03:06.834 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:03:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:07.119 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:03:07.135 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-16T04:03:07.076804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":635604,"kb_used_data":32408,"kb_used_omap":24,"kb_used_meta":602791,"kb_avail":1509313836,"statfs":{"total":1546188226560,"available":1545537368064,"internally_reserved":0,"allocated":33185792,"data_stored":19768750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":617258065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001813"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":425,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:16.104902+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T04:02:16.104676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:02:16.104676+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:30.120742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"28'1","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.558640+0000","last_change":"2024-09-16T04:02:13.919586+0000","last_active":"2024-09-16T04:03:03.558640+0000","last_peered":"2024-09-16T04:03:03.558640+0000","last_clean":"2024-09-16T04:03:03.558640+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:03:03.558640+0000","last_undegraded":"2024-09-16T04:03:03.558640+0000","last_fullsized":"2024-09-16T04:03:03.558640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'1","last_scrub_stamp":"2024-09-16T04:02:13.919386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:13.919386+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:39.323146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185421+0000","last_change":"2024-09-16T04:02:31.973520+0000","last_active":"2024-09-16T04:03:03.185421+0000","last_peered":"2024-09-16T04:03:03.185421+0000","last_clean":"2024-09-16T04:03:03.185421+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:03:03.185421+0000","last_undegraded":"2024-09-16T04:03:03.185421+0000","last_fullsized":"2024-09-16T04:03:03.185421+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.973151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.973151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:49.066570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":227,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137879+0000","last_change":"2024-09-16T04:02:30.059627+0000","last_active":"2024-09-16T04:03:04.137879+0000","last_peered":"2024-09-16T04:03:04.137879+0000","last_clean":"2024-09-16T04:03:04.137879+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:03:04.137879+0000","last_undegraded":"2024-09-16T04:03:04.137879+0000","last_fullsized":"2024-09-16T04:03:04.137879+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:30.059406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:30.059406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:57.455661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"105'3","reported_seq":225,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185459+0000","last_change":"2024-09-16T04:02:37.020433+0000","last_active":"2024-09-16T04:03:03.185459+0000","last_peered":"2024-09-16T04:03:03.185459+0000","last_clean":"2024-09-16T04:03:03.185459+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:03:03.185459+0000","last_undegraded":"2024-09-16T04:03:03.185459+0000","last_fullsized":"2024-09-16T04:03:03.185459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_clean_scrub_stamp":"2024-09-16T04:02:37.020087+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:45.933828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.185254+0000","last_change":"2024-09-16T04:02:31.948942+0000","last_active":"2024-09-16T04:03:03.185254+0000","last_peered":"2024-09-16T04:03:03.185254+0000","last_clean":"2024-09-16T04:03:03.185254+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:03:03.185254+0000","last_undegraded":"2024-09-16T04:03:03.185254+0000","last_fullsized":"2024-09-16T04:03:03.185254+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:31.948810+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.948810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:00.770248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.873784+0000","last_change":"2024-09-16T04:02:26.870604+0000","last_active":"2024-09-16T04:03:03.873784+0000","last_peered":"2024-09-16T04:03:03.873784+0000","last_clean":"2024-09-16T04:03:03.873784+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:03:03.873784+0000","last_undegraded":"2024-09-16T04:03:03.873784+0000","last_fullsized":"2024-09-16T04:03:03.873784+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.870379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.870379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:35.995441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":226,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:03.872222+0000","last_change":"2024-09-16T04:02:36.511128+0000","last_active":"2024-09-16T04:03:03.872222+0000","last_peered":"2024-09-16T04:03:03.872222+0000","last_clean":"2024-09-16T04:03:03.872222+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:03:03.872222+0000","last_undegraded":"2024-09-16T04:03:03.872222+0000","last_fullsized":"2024-09-16T04:03:03.872222+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:02:36.510902+0000","last_clean_scrub_stamp":"2024-09-16T04:02:36.510902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:55.939407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":224,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-16T04:03:04.137419+0000","last_change":"2024-09-16T04:02:26.602127+0000","last_active":"2024-09-16T04:03:04.137419+0000","last_peered":"2024-09-16T04:03:04.137419+0000","last_clean":"2024-09-16T04:03:04.137419+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:03:04.137419+0000","last_undegraded":"2024-09-16T04:03:04.137419+0000","last_fullsized":"2024-09-16T04:03:04.137419+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.601970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.601970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:46.647581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51204,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320636,"statfs":{"total":96636764160,"available":96584331264,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215135,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51208,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320632,"statfs":{"total":96636764160,"available":96584327168,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33156,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51212,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320628,"statfs":{"total":96636764160,"available":96584323072,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,3,5,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28996,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342844,"statfs":{"total":96636764160,"available":96607072256,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33156,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29000,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342840,"statfs":{"total":96636764160,"available":96607068160,"internally_reserved":0,"allocated":2015232,"data_stored":1171856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28996,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342844,"statfs":{"total":96636764160,"available":96607072256,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51720,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320120,"statfs":{"total":96636764160,"available":96583802880,"internally_reserved":0,"allocated":2543616,"data_stored":1700765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51184,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320656,"statfs":{"total":96636764160,"available":96584351744,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2527232,"data_stored":1691131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33160,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338680,"statfs":{"total":96636764160,"available":96602808320,"internally_reserved":0,"allocated":2015232,"data_stored":1171856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55364,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316476,"statfs":{"total":96636764160,"available":96580071424,"internally_reserved":0,"allocated":2011136,"data_stored":1171837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247836,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33136,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1994752,"data_stored":1162203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:03:07.136 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:03:07.434 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:07.434 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":134,"fsid":"e6b3d5c9-0596-44ca-b271-bebb6e706efc","created":"2024-09-16T04:00:41.999461+0000","modified":"2024-09-16T04:03:07.195454+0000","last_up_change":"2024-09-16T04:00:57.414587+0000","last_in_change":"2024-09-16T04:00:46.572321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:00:57.133339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:01:05.007470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"841234ee-1773-46d0-a547-ee692a129261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6817","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6819","nonce":3422299298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6824","nonce":3422299298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6831","nonce":3422299298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":3422299298},{"type":"v1","addr":"172.21.15.37:6829","nonce":3422299298}]},"public_addr":"172.21.15.37:6819/3422299298","cluster_addr":"172.21.15.37:6824/3422299298","heartbeat_back_addr":"172.21.15.37:6831/3422299298","heartbeat_front_addr":"172.21.15.37:6829/3422299298","state":["exists","up"]},{"osd":1,"uuid":"7245a6e4-d25d-406d-9ed1-47ef0c8c1078","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6809","nonce":1856047451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6811","nonce":1856047451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6815","nonce":1856047451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1856047451},{"type":"v1","addr":"172.21.15.39:6813","nonce":1856047451}]},"public_addr":"172.21.15.39:6809/1856047451","cluster_addr":"172.21.15.39:6811/1856047451","heartbeat_back_addr":"172.21.15.39:6815/1856047451","heartbeat_front_addr":"172.21.15.39:6813/1856047451","state":["exists","up"]},{"osd":2,"uuid":"8605b852-95b1-4488-b860-a04e018a9d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6817","nonce":3343342478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6819","nonce":3343342478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6823","nonce":3343342478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3343342478},{"type":"v1","addr":"172.21.15.117:6821","nonce":3343342478}]},"public_addr":"172.21.15.117:6817/3343342478","cluster_addr":"172.21.15.117:6819/3343342478","heartbeat_back_addr":"172.21.15.117:6823/3343342478","heartbeat_front_addr":"172.21.15.117:6821/3343342478","state":["exists","up"]},{"osd":3,"uuid":"9457ba25-d686-4f3b-8b67-6db4aed0af99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6803","nonce":2905616722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6805","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6807","nonce":2905616722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6816","nonce":2905616722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6809","nonce":2905616722},{"type":"v1","addr":"172.21.15.188:6811","nonce":2905616722}]},"public_addr":"172.21.15.188:6803/2905616722","cluster_addr":"172.21.15.188:6807/2905616722","heartbeat_back_addr":"172.21.15.188:6816/2905616722","heartbeat_front_addr":"172.21.15.188:6811/2905616722","state":["exists","up"]},{"osd":4,"uuid":"23118253-3204-403c-b58e-c9cf18689a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6801","nonce":2698627343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6803","nonce":2698627343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6807","nonce":2698627343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2698627343},{"type":"v1","addr":"172.21.15.37:6805","nonce":2698627343}]},"public_addr":"172.21.15.37:6801/2698627343","cluster_addr":"172.21.15.37:6803/2698627343","heartbeat_back_addr":"172.21.15.37:6807/2698627343","heartbeat_front_addr":"172.21.15.37:6805/2698627343","state":["exists","up"]},{"osd":5,"uuid":"8ed21a9c-c9d4-454b-ab83-c14538a08a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6825","nonce":2772028538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6827","nonce":2772028538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6831","nonce":2772028538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2772028538},{"type":"v1","addr":"172.21.15.39:6829","nonce":2772028538}]},"public_addr":"172.21.15.39:6825/2772028538","cluster_addr":"172.21.15.39:6827/2772028538","heartbeat_back_addr":"172.21.15.39:6831/2772028538","heartbeat_front_addr":"172.21.15.39:6829/2772028538","state":["exists","up"]},{"osd":6,"uuid":"fc4474a6-228a-430e-a7f6-75c86bb7b514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6801","nonce":3124837418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6803","nonce":3124837418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6807","nonce":3124837418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3124837418},{"type":"v1","addr":"172.21.15.117:6805","nonce":3124837418}]},"public_addr":"172.21.15.117:6801/3124837418","cluster_addr":"172.21.15.117:6803/3124837418","heartbeat_back_addr":"172.21.15.117:6807/3124837418","heartbeat_front_addr":"172.21.15.117:6805/3124837418","state":["exists","up"]},{"osd":7,"uuid":"fb4131d0-f4b4-44bf-a88f-832b545f817e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6806","nonce":4221150049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6810","nonce":4221150049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6815","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6817","nonce":4221150049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":4221150049},{"type":"v1","addr":"172.21.15.188:6813","nonce":4221150049}]},"public_addr":"172.21.15.188:6806/4221150049","cluster_addr":"172.21.15.188:6810/4221150049","heartbeat_back_addr":"172.21.15.188:6817/4221150049","heartbeat_front_addr":"172.21.15.188:6813/4221150049","state":["exists","up"]},{"osd":8,"uuid":"55467173-0628-4162-8d2f-fb76319de849","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6809","nonce":3581510146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6811","nonce":3581510146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6815","nonce":3581510146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":3581510146},{"type":"v1","addr":"172.21.15.37:6813","nonce":3581510146}]},"public_addr":"172.21.15.37:6809/3581510146","cluster_addr":"172.21.15.37:6811/3581510146","heartbeat_back_addr":"172.21.15.37:6815/3581510146","heartbeat_front_addr":"172.21.15.37:6813/3581510146","state":["exists","up"]},{"osd":9,"uuid":"66db6afc-248e-4197-a30b-2bc766b6b900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6801","nonce":2726119652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6803","nonce":2726119652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6807","nonce":2726119652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2726119652},{"type":"v1","addr":"172.21.15.39:6805","nonce":2726119652}]},"public_addr":"172.21.15.39:6801/2726119652","cluster_addr":"172.21.15.39:6803/2726119652","heartbeat_back_addr":"172.21.15.39:6807/2726119652","heartbeat_front_addr":"172.21.15.39:6805/2726119652","state":["exists","up"]},{"osd":10,"uuid":"54464474-a85c-4a44-8c78-ca3439d56cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6825","nonce":737103121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6827","nonce":737103121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6831","nonce":737103121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":737103121},{"type":"v1","addr":"172.21.15.117:6829","nonce":737103121}]},"public_addr":"172.21.15.117:6825/737103121","cluster_addr":"172.21.15.117:6827/737103121","heartbeat_back_addr":"172.21.15.117:6831/737103121","heartbeat_front_addr":"172.21.15.117:6829/737103121","state":["exists","up"]},{"osd":11,"uuid":"9dabcd8b-28a8-43b2-9636-e3bb7d7a6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6823","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6824","nonce":1355328456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6828","nonce":1355328456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6833","nonce":1355328456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1355328456},{"type":"v1","addr":"172.21.15.188:6831","nonce":1355328456}]},"public_addr":"172.21.15.188:6824/1355328456","cluster_addr":"172.21.15.188:6828/1355328456","heartbeat_back_addr":"172.21.15.188:6833/1355328456","heartbeat_front_addr":"172.21.15.188:6831/1355328456","state":["exists","up"]},{"osd":12,"uuid":"43465856-0b63-41ca-b5e5-57771dbfbe5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6818","nonce":3693386273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6821","nonce":3693386273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6827","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6828","nonce":3693386273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3693386273},{"type":"v1","addr":"172.21.15.37:6825","nonce":3693386273}]},"public_addr":"172.21.15.37:6818/3693386273","cluster_addr":"172.21.15.37:6821/3693386273","heartbeat_back_addr":"172.21.15.37:6828/3693386273","heartbeat_front_addr":"172.21.15.37:6825/3693386273","state":["exists","up"]},{"osd":13,"uuid":"7fba1ed8-6948-4b3d-afbf-236c93b9e3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6817","nonce":2905392027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6819","nonce":2905392027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6823","nonce":2905392027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2905392027},{"type":"v1","addr":"172.21.15.39:6821","nonce":2905392027}]},"public_addr":"172.21.15.39:6817/2905392027","cluster_addr":"172.21.15.39:6819/2905392027","heartbeat_back_addr":"172.21.15.39:6823/2905392027","heartbeat_front_addr":"172.21.15.39:6821/2905392027","state":["exists","up"]},{"osd":14,"uuid":"5073cd26-fa48-45c5-ba56-3b0c20878c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6809","nonce":1375514506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6811","nonce":1375514506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6815","nonce":1375514506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1375514506},{"type":"v1","addr":"172.21.15.117:6813","nonce":1375514506}]},"public_addr":"172.21.15.117:6809/1375514506","cluster_addr":"172.21.15.117:6811/1375514506","heartbeat_back_addr":"172.21.15.117:6815/1375514506","heartbeat_front_addr":"172.21.15.117:6813/1375514506","state":["exists","up"]},{"osd":15,"uuid":"52f9cbf1-4411-47e5-a8e9-442f99215ee3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6819","nonce":3000701187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6821","nonce":3000701187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6829","nonce":3000701187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3000701187},{"type":"v1","addr":"172.21.15.188:6825","nonce":3000701187}]},"public_addr":"172.21.15.188:6819/3000701187","cluster_addr":"172.21.15.188:6821/3000701187","heartbeat_back_addr":"172.21.15.188:6829/3000701187","heartbeat_front_addr":"172.21.15.188:6825/3000701187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.245592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:03:07.058631+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:06.602619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:58.884947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:15.931876+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:54.754303+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:03:06.541044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:01.375477+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:01:57.934680+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:17.873726+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:57.781946+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:16.811116+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.982880+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:21.914975+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:08.616813+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:02:06.605685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":17,"length":1},{"begin":30,"length":1},{"begin":35,"length":1},{"begin":38,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:03:08.452 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T04:03:08.453 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:08.616 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:08.616 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:08.616 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:08.632 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T04:03:08.928 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T04:03:08.947 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T04:03:08.948 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:09.108 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:09.108 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:09.109 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:09.126 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T04:03:09.425 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T04:03:09.445 INFO:tasks.ceph:Scrubbing osd.8 2024-09-16T04:03:09.445 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:09.592 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:09.592 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:09.592 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:09.608 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-16T04:03:09.902 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 8 to deep-scrub 2024-09-16T04:03:09.920 INFO:tasks.ceph:Scrubbing osd.12 2024-09-16T04:03:09.921 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:10.072 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:10.072 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:10.072 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:10.089 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-16T04:03:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 12 to deep-scrub 2024-09-16T04:03:10.404 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T04:03:10.405 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:10.562 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:10.562 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:10.562 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:10.578 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T04:03:10.863 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T04:03:10.882 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T04:03:10.882 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:11.031 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:11.031 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:11.046 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T04:03:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T04:03:11.371 INFO:tasks.ceph:Scrubbing osd.9 2024-09-16T04:03:11.371 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:11.519 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:11.519 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:11.519 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:11.535 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-16T04:03:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 9 to deep-scrub 2024-09-16T04:03:11.849 INFO:tasks.ceph:Scrubbing osd.13 2024-09-16T04:03:11.849 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:12.011 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:12.011 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:12.011 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:12.028 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-16T04:03:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 13 to deep-scrub 2024-09-16T04:03:12.348 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T04:03:12.348 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:12.499 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:12.499 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:12.499 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:12.514 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T04:03:12.799 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T04:03:12.817 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T04:03:12.817 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:12.961 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:12.961 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:12.961 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:12.975 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T04:03:13.262 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T04:03:13.280 INFO:tasks.ceph:Scrubbing osd.10 2024-09-16T04:03:13.280 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:13.434 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:13.434 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:13.450 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-16T04:03:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 10 to deep-scrub 2024-09-16T04:03:13.759 INFO:tasks.ceph:Scrubbing osd.14 2024-09-16T04:03:13.759 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:13.907 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:13.908 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:13.908 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:13.922 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-16T04:03:14.223 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 14 to deep-scrub 2024-09-16T04:03:14.243 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T04:03:14.243 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:14.402 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:14.403 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:14.403 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:14.419 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T04:03:14.717 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T04:03:14.737 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T04:03:14.737 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:14.890 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:14.890 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:14.890 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:14.907 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T04:03:15.203 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T04:03:15.223 INFO:tasks.ceph:Scrubbing osd.11 2024-09-16T04:03:15.223 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:15.382 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:15.382 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:15.382 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:15.398 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-16T04:03:15.700 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 11 to deep-scrub 2024-09-16T04:03:15.719 INFO:tasks.ceph:Scrubbing osd.15 2024-09-16T04:03:15.720 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:03:15.864 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T04:03:15.864 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:03:15.864 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T04:03:15.880 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-16T04:03:16.184 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 15 to deep-scrub 2024-09-16T04:03:16.203 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:03:16.497 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:16.498 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:03:16.514 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-16T04:03:15.079110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":636168,"kb_used_data":33016,"kb_used_omap":24,"kb_used_meta":602791,"kb_avail":1509313272,"statfs":{"total":1546188226560,"available":1545536790528,"internally_reserved":0,"allocated":33808384,"data_stored":20137070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":617258065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.862566"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":432,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T04:03:12.243395+0000","last_change":"2024-09-16T04:02:16.104902+0000","last_active":"2024-09-16T04:03:12.243395+0000","last_peered":"2024-09-16T04:03:12.243395+0000","last_clean":"2024-09-16T04:03:12.243395+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:03:12.243395+0000","last_undegraded":"2024-09-16T04:03:12.243395+0000","last_fullsized":"2024-09-16T04:03:12.243395+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T04:02:16.104676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:00:57.414587+0000","last_clean_scrub_stamp":"2024-09-16T04:02:16.104676+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"135'2","reported_seq":241,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T04:03:12.308523+0000","last_change":"2024-09-16T04:03:12.308523+0000","last_active":"2024-09-16T04:03:12.308523+0000","last_peered":"2024-09-16T04:03:12.308523+0000","last_clean":"2024-09-16T04:03:12.308523+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:03:12.308523+0000","last_undegraded":"2024-09-16T04:03:12.308523+0000","last_fullsized":"2024-09-16T04:03:12.308523+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'2","last_scrub_stamp":"2024-09-16T04:03:12.308244+0000","last_deep_scrub":"135'2","last_deep_scrub_stamp":"2024-09-16T04:03:12.308244+0000","last_clean_scrub_stamp":"2024-09-16T04:03:12.308244+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:33.376268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":227,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T04:03:08.473859+0000","last_change":"2024-09-16T04:02:31.973520+0000","last_active":"2024-09-16T04:03:08.473859+0000","last_peered":"2024-09-16T04:03:08.473859+0000","last_clean":"2024-09-16T04:03:08.473859+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:03:08.473859+0000","last_undegraded":"2024-09-16T04:03:08.473859+0000","last_fullsized":"2024-09-16T04:03:08.473859+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:31.973151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:31.973151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:49.066570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":235,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T04:03:12.328986+0000","last_change":"2024-09-16T04:02:30.059627+0000","last_active":"2024-09-16T04:03:12.328986+0000","last_peered":"2024-09-16T04:03:12.328986+0000","last_clean":"2024-09-16T04:03:12.328986+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:03:12.328986+0000","last_undegraded":"2024-09-16T04:03:12.328986+0000","last_fullsized":"2024-09-16T04:03:12.328986+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:30.059406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:30.059406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"105'3","reported_seq":230,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T04:03:08.473951+0000","last_change":"2024-09-16T04:02:37.020433+0000","last_active":"2024-09-16T04:03:08.473951+0000","last_peered":"2024-09-16T04:03:08.473951+0000","last_clean":"2024-09-16T04:03:08.473951+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:03:08.473951+0000","last_undegraded":"2024-09-16T04:03:08.473951+0000","last_fullsized":"2024-09-16T04:03:08.473951+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-16T04:02:37.020087+0000","last_clean_scrub_stamp":"2024-09-16T04:02:37.020087+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:45.933828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":240,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T04:03:10.756619+0000","last_change":"2024-09-16T04:03:10.756619+0000","last_active":"2024-09-16T04:03:10.756619+0000","last_peered":"2024-09-16T04:03:10.756619+0000","last_clean":"2024-09-16T04:03:10.756619+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:03:10.756619+0000","last_undegraded":"2024-09-16T04:03:10.756619+0000","last_fullsized":"2024-09-16T04:03:10.756619+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:03:10.756467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:03:10.756467+0000","last_clean_scrub_stamp":"2024-09-16T04:03:10.756467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:15.083109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":233,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T04:03:12.615680+0000","last_change":"2024-09-16T04:02:26.870604+0000","last_active":"2024-09-16T04:03:12.615680+0000","last_peered":"2024-09-16T04:03:12.615680+0000","last_clean":"2024-09-16T04:03:12.615680+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:03:12.615680+0000","last_undegraded":"2024-09-16T04:03:12.615680+0000","last_fullsized":"2024-09-16T04:03:12.615680+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.870379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.870379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:35.995441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":239,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T04:03:10.535367+0000","last_change":"2024-09-16T04:03:10.535408+0000","last_active":"2024-09-16T04:03:10.535367+0000","last_peered":"2024-09-16T04:03:10.535367+0000","last_clean":"2024-09-16T04:03:10.535367+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:03:10.535367+0000","last_undegraded":"2024-09-16T04:03:10.535367+0000","last_fullsized":"2024-09-16T04:03:10.535367+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:03:10.535268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:03:10.535268+0000","last_clean_scrub_stamp":"2024-09-16T04:03:10.535268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:33.175978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":228,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T04:03:09.232024+0000","last_change":"2024-09-16T04:02:26.602127+0000","last_active":"2024-09-16T04:03:09.232024+0000","last_peered":"2024-09-16T04:03:09.232024+0000","last_clean":"2024-09-16T04:03:09.232024+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:03:09.232024+0000","last_undegraded":"2024-09-16T04:03:09.232024+0000","last_fullsized":"2024-09-16T04:03:09.232024+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:02:26.601970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:01:05.697312+0000","last_clean_scrub_stamp":"2024-09-16T04:02:26.601970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:03:46.647581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215136,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33184,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338656,"statfs":{"total":96636764160,"available":96602783744,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215136,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51232,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320608,"statfs":{"total":96636764160,"available":96584302592,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215137,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51248,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320592,"statfs":{"total":96636764160,"available":96584286208,"internally_reserved":0,"allocated":2060288,"data_stored":1200915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33188,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338652,"statfs":{"total":96636764160,"available":96602779648,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51232,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320608,"statfs":{"total":96636764160,"available":96584302592,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,3,5,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33184,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338656,"statfs":{"total":96636764160,"available":96602783744,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29024,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342816,"statfs":{"total":96636764160,"available":96607043584,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33188,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338652,"statfs":{"total":96636764160,"available":96602779648,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29028,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342812,"statfs":{"total":96636764160,"available":96607039488,"internally_reserved":0,"allocated":2048000,"data_stored":1191258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29024,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342816,"statfs":{"total":96636764160,"available":96607043584,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215136,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51764,"kb_used_data":2532,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":2592768,"data_stored":1729843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51232,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320608,"statfs":{"total":96636764160,"available":96584302592,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37868,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333972,"statfs":{"total":96636764160,"available":96597987328,"internally_reserved":0,"allocated":2576384,"data_stored":1720167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247840,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2048000,"data_stored":1191258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247840,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55396,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316444,"statfs":{"total":96636764160,"available":96580038656,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33184,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338656,"statfs":{"total":96636764160,"available":96602783744,"internally_reserved":0,"allocated":2043904,"data_stored":1191239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:03:16.519 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T04:02:16.104676+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=2, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=3, tm_sec=7, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:03:16.519 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T04:02:31.973151+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=2, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=3, tm_sec=7, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:03:16.519 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T04:02:30.059406+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=2, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=3, tm_sec=7, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:03:16.519 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T04:02:37.020087+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=2, tm_sec=37, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=3, tm_sec=7, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:03:16.519 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T04:02:26.870379+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=2, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=3, tm_sec=7, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:03:16.519 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T04:02:26.601970+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=2, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=3, tm_sec=7, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:03:16.519 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T04:03:36.521 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:03:36.834 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T04:03:36.835 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T04:03:36.853 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-16T04:03:35.085054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":638100,"kb_used_data":34952,"kb_used_omap":24,"kb_used_meta":602791,"kb_avail":1509311340,"statfs":{"total":1546188226560,"available":1545534812160,"internally_reserved":0,"allocated":35790848,"data_stored":21307866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":617258065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002433"},"pg_stats":[{"pgid":"1.0","version":"136'159","reported_seq":456,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T04:03:31.078366+0000","last_change":"2024-09-16T04:03:12.976406+0000","last_active":"2024-09-16T04:03:31.078366+0000","last_peered":"2024-09-16T04:03:31.078366+0000","last_clean":"2024-09-16T04:03:31.078366+0000","last_became_active":"2024-09-16T04:00:59.013974+0000","last_became_peered":"2024-09-16T04:00:59.013974+0000","last_unstale":"2024-09-16T04:03:31.078366+0000","last_undegraded":"2024-09-16T04:03:31.078366+0000","last_fullsized":"2024-09-16T04:03:31.078366+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'159","last_scrub_stamp":"2024-09-16T04:03:12.976158+0000","last_deep_scrub":"136'159","last_deep_scrub_stamp":"2024-09-16T04:03:12.976158+0000","last_clean_scrub_stamp":"2024-09-16T04:03:12.976158+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:26.151000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"135'2","reported_seq":257,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T04:03:30.766444+0000","last_change":"2024-09-16T04:03:12.308523+0000","last_active":"2024-09-16T04:03:30.766444+0000","last_peered":"2024-09-16T04:03:30.766444+0000","last_clean":"2024-09-16T04:03:30.766444+0000","last_became_active":"2024-09-16T04:01:06.729693+0000","last_became_peered":"2024-09-16T04:01:06.729693+0000","last_unstale":"2024-09-16T04:03:30.766444+0000","last_undegraded":"2024-09-16T04:03:30.766444+0000","last_fullsized":"2024-09-16T04:03:30.766444+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'2","last_scrub_stamp":"2024-09-16T04:03:12.308244+0000","last_deep_scrub":"135'2","last_deep_scrub_stamp":"2024-09-16T04:03:12.308244+0000","last_clean_scrub_stamp":"2024-09-16T04:03:12.308244+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:33.376268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084485300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":251,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T04:03:31.266941+0000","last_change":"2024-09-16T04:03:15.005065+0000","last_active":"2024-09-16T04:03:31.266941+0000","last_peered":"2024-09-16T04:03:31.266941+0000","last_clean":"2024-09-16T04:03:31.266941+0000","last_became_active":"2024-09-16T04:01:06.723233+0000","last_became_peered":"2024-09-16T04:01:06.723233+0000","last_unstale":"2024-09-16T04:03:31.266941+0000","last_undegraded":"2024-09-16T04:03:31.266941+0000","last_fullsized":"2024-09-16T04:03:31.266941+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:03:15.004802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:03:15.004802+0000","last_clean_scrub_stamp":"2024-09-16T04:03:15.004802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:28.526965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080475000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":257,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T04:03:31.078436+0000","last_change":"2024-09-16T04:03:14.008614+0000","last_active":"2024-09-16T04:03:31.078436+0000","last_peered":"2024-09-16T04:03:31.078436+0000","last_clean":"2024-09-16T04:03:31.078436+0000","last_became_active":"2024-09-16T04:01:06.727627+0000","last_became_peered":"2024-09-16T04:01:06.727627+0000","last_unstale":"2024-09-16T04:03:31.078436+0000","last_undegraded":"2024-09-16T04:03:31.078436+0000","last_fullsized":"2024-09-16T04:03:31.078436+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:03:14.008374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:03:14.008374+0000","last_clean_scrub_stamp":"2024-09-16T04:03:14.008374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:24.270718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011140360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"105'3","reported_seq":254,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T04:03:31.266849+0000","last_change":"2024-09-16T04:03:16.053694+0000","last_active":"2024-09-16T04:03:31.266849+0000","last_peered":"2024-09-16T04:03:31.266849+0000","last_clean":"2024-09-16T04:03:31.266849+0000","last_became_active":"2024-09-16T04:01:06.726845+0000","last_became_peered":"2024-09-16T04:01:06.726845+0000","last_unstale":"2024-09-16T04:03:31.266849+0000","last_undegraded":"2024-09-16T04:03:31.266849+0000","last_fullsized":"2024-09-16T04:03:31.266849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-16T04:03:16.053464+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-16T04:03:16.053464+0000","last_clean_scrub_stamp":"2024-09-16T04:03:16.053464+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:18.825236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020172720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":255,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T04:03:31.267177+0000","last_change":"2024-09-16T04:03:10.756619+0000","last_active":"2024-09-16T04:03:31.267177+0000","last_peered":"2024-09-16T04:03:31.267177+0000","last_clean":"2024-09-16T04:03:31.267177+0000","last_became_active":"2024-09-16T04:01:07.046238+0000","last_became_peered":"2024-09-16T04:01:07.046238+0000","last_unstale":"2024-09-16T04:03:31.267177+0000","last_undegraded":"2024-09-16T04:03:31.267177+0000","last_fullsized":"2024-09-16T04:03:31.267177+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:03:10.756467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:03:10.756467+0000","last_clean_scrub_stamp":"2024-09-16T04:03:10.756467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:15.083109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073365099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":255,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T04:03:31.267606+0000","last_change":"2024-09-16T04:03:13.953398+0000","last_active":"2024-09-16T04:03:31.267606+0000","last_peered":"2024-09-16T04:03:31.267606+0000","last_clean":"2024-09-16T04:03:31.267606+0000","last_became_active":"2024-09-16T04:01:06.723653+0000","last_became_peered":"2024-09-16T04:01:06.723653+0000","last_unstale":"2024-09-16T04:03:31.267606+0000","last_undegraded":"2024-09-16T04:03:31.267606+0000","last_fullsized":"2024-09-16T04:03:31.267606+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:03:13.953265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:03:13.953265+0000","last_clean_scrub_stamp":"2024-09-16T04:03:13.953265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:31.448356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000963724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":253,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T04:03:27.350077+0000","last_change":"2024-09-16T04:03:10.535408+0000","last_active":"2024-09-16T04:03:27.350077+0000","last_peered":"2024-09-16T04:03:27.350077+0000","last_clean":"2024-09-16T04:03:27.350077+0000","last_became_active":"2024-09-16T04:01:06.720602+0000","last_became_peered":"2024-09-16T04:01:06.720602+0000","last_unstale":"2024-09-16T04:03:27.350077+0000","last_undegraded":"2024-09-16T04:03:27.350077+0000","last_fullsized":"2024-09-16T04:03:27.350077+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:03:10.535268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:03:10.535268+0000","last_clean_scrub_stamp":"2024-09-16T04:03:10.535268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:33.175978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095783200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":252,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T04:03:31.268626+0000","last_change":"2024-09-16T04:03:16.437012+0000","last_active":"2024-09-16T04:03:31.268626+0000","last_peered":"2024-09-16T04:03:31.268626+0000","last_clean":"2024-09-16T04:03:31.268626+0000","last_became_active":"2024-09-16T04:01:06.725081+0000","last_became_peered":"2024-09-16T04:01:06.725081+0000","last_unstale":"2024-09-16T04:03:31.268626+0000","last_undegraded":"2024-09-16T04:03:31.268626+0000","last_fullsized":"2024-09-16T04:03:31.268626+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:03:16.436767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:03:16.436767+0000","last_clean_scrub_stamp":"2024-09-16T04:03:16.436767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:40.869402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072781500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215140,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33312,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338528,"statfs":{"total":96636764160,"available":96602652672,"internally_reserved":0,"allocated":2174976,"data_stored":1268647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":24,"seq":103079215140,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51360,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320480,"statfs":{"total":96636764160,"available":96584171520,"internally_reserved":0,"allocated":2174976,"data_stored":1268647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":24,"seq":103079215141,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51360,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320480,"statfs":{"total":96636764160,"available":96584171520,"internally_reserved":0,"allocated":2174976,"data_stored":1268647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,4,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280549,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33296,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2158592,"data_stored":1258971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51352,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320488,"statfs":{"total":96636764160,"available":96584179712,"internally_reserved":0,"allocated":2158592,"data_stored":1258971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,3,5,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2158592,"data_stored":1258971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29152,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342688,"statfs":{"total":96636764160,"available":96606912512,"internally_reserved":0,"allocated":2174976,"data_stored":1268647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33296,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2158592,"data_stored":1258971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29156,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342684,"statfs":{"total":96636764160,"available":96606908416,"internally_reserved":0,"allocated":2179072,"data_stored":1268666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29136,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342704,"statfs":{"total":96636764160,"available":96606928896,"internally_reserved":0,"allocated":2158592,"data_stored":1258971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51884,"kb_used_data":2644,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319956,"statfs":{"total":96636764160,"available":96583634944,"internally_reserved":0,"allocated":2707456,"data_stored":1797575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51360,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320480,"statfs":{"total":96636764160,"available":96584171520,"internally_reserved":0,"allocated":2174976,"data_stored":1268647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37992,"kb_used_data":2644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333848,"statfs":{"total":96636764160,"available":96597860352,"internally_reserved":0,"allocated":2707456,"data_stored":1797575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247844,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33316,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":2179072,"data_stored":1268666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247844,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55520,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316320,"statfs":{"total":96636764160,"available":96579911680,"internally_reserved":0,"allocated":2174976,"data_stored":1268647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,6,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247842,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33312,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338528,"statfs":{"total":96636764160,"available":96602652672,"internally_reserved":0,"allocated":2174976,"data_stored":1268647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:03:36.854 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T04:03:37.188 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T04:03:37.188 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T04:03:37.188 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:03:37.189 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:37.256 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:03:37.256 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T04:03:37.256 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:37.317 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T04:03:37.318 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-16T04:03:37.318 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:37.403 INFO:tasks.ceph.osd.8:Stopped 2024-09-16T04:03:37.404 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-16T04:03:37.404 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:37.502 INFO:tasks.ceph.osd.12:Stopped 2024-09-16T04:03:37.502 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:03:37.502 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:37.577 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:03:37.578 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T04:03:37.578 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:37.671 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T04:03:37.671 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-16T04:03:37.671 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:40.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2024-09-16T04:03:43.772 INFO:tasks.ceph.osd.9:Stopped 2024-09-16T04:03:43.773 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-16T04:03:43.773 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:43.857 INFO:tasks.ceph.osd.13:Stopped 2024-09-16T04:03:43.858 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T04:03:43.858 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:43.936 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T04:03:43.936 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T04:03:43.936 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:44.023 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T04:03:44.023 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-16T04:03:44.023 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:44.111 INFO:tasks.ceph.osd.10:Stopped 2024-09-16T04:03:44.111 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-16T04:03:44.111 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:46.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 is failed for ~0s 2024-09-16T04:03:46.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 has been restored 2024-09-16T04:03:50.213 INFO:tasks.ceph.osd.14:Stopped 2024-09-16T04:03:50.213 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:03:50.213 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:50.299 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:03:50.299 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T04:03:50.299 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:50.380 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T04:03:50.380 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-16T04:03:50.381 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:51.831 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 is failed for ~0s 2024-09-16T04:03:51.831 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 has been restored 2024-09-16T04:03:56.482 INFO:tasks.ceph.osd.11:Stopped 2024-09-16T04:03:56.482 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-16T04:03:56.482 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:56.544 INFO:tasks.ceph.osd.15:Stopped 2024-09-16T04:03:56.544 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T04:03:56.545 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T04:03:56.545 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:56.573 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T04:03:56.573 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T04:03:56.573 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:56.613 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T04:03:56.613 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T04:03:56.614 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:03:56.614 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:56.632 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:03:56.633 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:03:56.633 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:56.658 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:03:56.658 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:03:56.658 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:56.676 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:03:56.676 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T04:03:56.676 DEBUG:teuthology.orchestra.run.smithi037:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-16T04:03:56.696 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T04:03:56.696 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T04:03:56.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 has been restored 2024-09-16T04:03:56.896 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T04:03:56.896 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T04:03:57.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T04:03:57.196 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-16T04:03:57.328 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T04:03:57.329 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-16T04:03:57.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T04:03:57.470 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T04:03:57.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T04:03:57.647 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T04:03:57.789 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T04:03:57.790 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-16T04:03:57.904 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T04:03:57.904 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-16T04:03:58.017 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:03:58.017 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T04:03:58.202 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:03:58.202 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T04:03:58.408 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:03:58.408 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-16T04:03:58.591 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:03:58.591 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-16T04:03:58.759 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:03:58.760 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T04:03:58.951 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:03:58.951 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T04:03:59.173 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:03:59.174 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-16T04:03:59.325 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:03:59.325 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-16T04:03:59.473 INFO:tasks.ceph:Archiving mon data... 2024-09-16T04:03:59.474 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/data/mon.a.tgz 2024-09-16T04:03:59.474 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-09-16T04:03:59.481 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.Nwxx29T9Wz 2024-09-16T04:03:59.481 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Nwxx29T9Wz 2024-09-16T04:03:59.632 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.Nwxx29T9Wz 2024-09-16T04:03:59.697 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.Nwxx29T9Wz is 422KB 2024-09-16T04:03:59.759 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.Nwxx29T9Wz 2024-09-16T04:03:59.765 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/data/mon.b.tgz 2024-09-16T04:03:59.765 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-16T04:03:59.771 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.jgLxy4G3a1 2024-09-16T04:03:59.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.jgLxy4G3a1 2024-09-16T04:03:59.957 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.jgLxy4G3a1 2024-09-16T04:04:00.022 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.jgLxy4G3a1 is 593KB 2024-09-16T04:04:00.088 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.jgLxy4G3a1 2024-09-16T04:04:00.096 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/data/mon.c.tgz 2024-09-16T04:04:00.096 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-09-16T04:04:00.102 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.NjkZoWRjpi 2024-09-16T04:04:00.102 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.NjkZoWRjpi 2024-09-16T04:04:00.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.NjkZoWRjpi 2024-09-16T04:04:00.351 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.NjkZoWRjpi is 593KB 2024-09-16T04:04:00.414 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.NjkZoWRjpi 2024-09-16T04:04:00.418 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T04:04:00.419 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:04:00.421 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:04:00.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:04:00.461 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:04:00.482 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:04:00.488 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:04:00.516 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:04:00.523 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:04:00.523 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T04:04:00.525 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi037/crash 2024-09-16T04:04:00.525 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:04:00.539 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi039/crash 2024-09-16T04:04:00.540 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:04:00.555 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi117/crash 2024-09-16T04:04:00.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:04:00.572 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi188/crash 2024-09-16T04:04:00.573 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:04:00.588 INFO:tasks.ceph:Compressing logs... 2024-09-16T04:04:00.588 DEBUG:teuthology.orchestra.run.smithi037:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:04:00.591 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:04:00.602 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:04:00.609 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35536.log 2024-09-16T04:04:00.609 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35035.log 2024-09-16T04:04:00.610 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35536.log.gz 2024-09-16T04:04:00.610 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35939.log 2024-09-16T04:04:00.612 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-09-16T04:04:00.612 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35035.log: /var/log/ceph/ceph-client.admin.35939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35841.log 2024-09-16T04:04:00.612 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35939.log.gz 2024-09-16T04:04:00.612 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35035.log.gz 2024-09-16T04:04:00.612 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-09-16T04:04:00.612 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36658.log 2024-09-16T04:04:00.614 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36996.log 2024-09-16T04:04:00.614 INFO:teuthology.orchestra.run.smithi037.stderr: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.35841.log.gz 2024-09-16T04:04:00.614 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36658.log.gz 2024-09-16T04:04:00.614 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36453.log 2024-09-16T04:04:00.614 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31344.log 2024-09-16T04:04:00.615 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36996.log.gz 2024-09-16T04:04:00.615 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36453.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T04:04:00.615 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36453.log.gz 2024-09-16T04:04:00.615 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39157.log 2024-09-16T04:04:00.615 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31344.log.gz 2024-09-16T04:04:00.615 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31333.log 2024-09-16T04:04:00.617 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32849.log 2024-09-16T04:04:00.617 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39157.log.gz 2024-09-16T04:04:00.617 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37952.log/var/log/ceph/ceph-client.admin.31333.log: 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31333.log.gz 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38687.log 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32849.log.gz 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35770.log 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37952.log.gz 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5/var/log/ceph/ceph-client.admin.38687.log: --verbose -- /var/log/ceph/ceph-client.admin.37603.log 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38687.log.gz 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35770.log.gz 2024-09-16T04:04:00.618 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39209.log 2024-09-16T04:04:00.619 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32947.log 2024-09-16T04:04:00.619 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37603.log.gz 2024-09-16T04:04:00.623 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39209.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39209.log.gz 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37465.log 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32947.log.gz 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36376.log 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34904.log 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37465.log.gz 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36376.log: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36376.log.gz 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31337.log 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38019.log 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34904.log.gz 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5/var/log/ceph/ceph-client.admin.31337.log: --verbose -- /var/log/ceph/ceph-client.admin.35493.log 2024-09-16T04:04:00.624 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31337.log.gz 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.38019.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31403.log 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38019.log.gz 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33358.log 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.35493.log.gz 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30939.log 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31403.log.gz 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.33358.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30962.log 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr: 27.8% -- replaced with /var/log/ceph/ceph-client.admin.33358.log.gz 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39025.log 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30939.log.gz 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33062.log 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30962.log.gz 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38427.log 2024-09-16T04:04:00.625 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39025.log.gz 2024-09-16T04:04:00.626 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38765.log 2024-09-16T04:04:00.626 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33062.log.gz 2024-09-16T04:04:00.626 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38427.log: -- /var/log/ceph/ceph-client.admin.37571.log 2024-09-16T04:04:00.626 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38427.log.gz 2024-09-16T04:04:00.629 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T04:04:00.629 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T04:04:00.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:04:00.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:04:00.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-16T04:04:00.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-16T04:04:00.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T04:04:00.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph-mon.b.log: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:04:00.630 INFO:teuthology.orchestra.run.smithi039.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:04:00.633 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:04:00.641 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34074.log 2024-09-16T04:04:00.641 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38765.log.gz 2024-09-16T04:04:00.641 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34138.log 2024-09-16T04:04:00.641 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37571.log.gz 2024-09-16T04:04:00.641 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37595.log 2024-09-16T04:04:00.641 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34074.log.gz 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31383.log 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34138.log.gz 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37595.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.37387.log -- replaced with /var/log/ceph/ceph-client.admin.37595.log.gz 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33758.log 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31383.log.gz 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36627.log 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37387.log.gz 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32257.log 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33758.log.gz 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32360.log 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36627.log.gz 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.32257.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-09-16T04:04:00.642 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32257.log.gz 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32641.log 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32360.log.gz 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31878.log 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5/var/log/ceph/ceph-client.admin.32641.log: --verbose -- /var/log/ceph/ceph-client.admin.39415.log 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32641.log.gz 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38022.log 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31878.log.gz 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31737.log 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39415.log.gz 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36034.log 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38022.log.gz 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34905.log 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31737.log.gz 2024-09-16T04:04:00.643 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34186.log 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36034.log.gz 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30930.log 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34905.log.gz 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39105.log 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34186.log.gz 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37597.log 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30930.log.gz 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34794.log 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39105.log.gz 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37597.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37597.log.gz -5 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36347.log 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34794.log.gz 2024-09-16T04:04:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36918.log 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36347.log.gz 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37022.log 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33916.log 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36918.log.gz 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37022.log.gz 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34639.log 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33227.log 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33916.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.33916.log.gz 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34639.log.gz 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38999.log 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37491.log 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33227.log.gz 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34215.log 2024-09-16T04:04:00.645 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38999.log.gz 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36575.log 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37491.log.gz 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37601.log 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.34215.log.gz 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36575.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.26517.log -- replaced with /var/log/ceph/ceph-client.admin.36575.log.gz 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37601.log.gz 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34395.log 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36788.log 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26517.log.gz 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34395.log: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.34395.log.gz 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33223.log 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36788.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36788.log.gz 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31777.log 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33223.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33223.log.gz -5 2024-09-16T04:04:00.646 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30935.log 2024-09-16T04:04:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37594.log 2024-09-16T04:04:00.647 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31777.log.gz 2024-09-16T04:04:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30935.log.gz 2024-09-16T04:04:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37413.log 2024-09-16T04:04:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-09-16T04:04:00.647 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37594.log.gz 2024-09-16T04:04:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37413.log.gz 2024-09-16T04:04:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2024-09-16T04:04:00.651 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T04:04:00.651 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T04:04:00.651 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-16T04:04:00.651 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:04:00.651 INFO:teuthology.orchestra.run.smithi117.stderr:gzip/var/log/ceph/ceph-osd.14.log: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T04:04:00.651 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:04:00.651 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-16T04:04:00.651 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.audit.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:04:00.652 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.10.log: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:04:00.670 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26318.log 2024-09-16T04:04:00.670 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26270.log 2024-09-16T04:04:00.670 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26318.log.gz 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26366.log 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26270.log.gz 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31615.log 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26366.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26517.log 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26366.log.gz 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.0.31615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26469.log 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr: 53.4% -- replaced with /var/log/ceph/ceph-client.0.31615.log.gz 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26517.log.gz 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26590.log 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26469.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26469.log.gz 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26390.log 2024-09-16T04:04:00.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26590.log.gz 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26414.log 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26390.log.gz 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26342.log 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26414.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26414.log.gz 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26342.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26342.log.gz 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26493.log 2024-09-16T04:04:00.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26541.log 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26566.log 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26493.log.gz 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26294.log 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26541.log.gz 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26445.log 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26566.log.gz 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26246.log 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26294.log.gz 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26614.log 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26445.log.gz 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26246.log.gz 2024-09-16T04:04:00.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.26614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26614.log.gz 2024-09-16T04:04:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34504.log 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39131.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34504.log.gz 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33177.log 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37572.log 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38012.log 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33177.log.gz 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37572.log.gz 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32979.log 2024-09-16T04:04:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31347.log 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38012.log: /var/log/ceph/ceph-client.admin.32979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38012.log.gz 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.32979.log.gz 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37335.log 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30929.log 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31347.log: /var/log/ceph/ceph-client.admin.37335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31347.log.gz 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34718.log 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37335.log.gz 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30929.log.gz 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36526.log 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35733.log 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34718.log: /var/log/ceph/ceph-client.admin.36526.log: 25.2% -- replaced with /var/log/ceph/ceph-client.admin.34718.log.gz 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37950.log 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36526.log.gz 2024-09-16T04:04:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32354.log 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr: 25.0% -- replaced with /var/log/ceph/ceph-client.admin.35733.log.gz 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31854.log 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37950.log.gz 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32354.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.32354.log.gz 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38609.log 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34983.log 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31854.log.gz 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33299.log 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38609.log.gz 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34983.log.gz 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36275.log 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37578.log 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33299.log.gz 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32508.log 2024-09-16T04:04:00.721 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36275.log.gz 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37578.log.gz 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33690.log 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32313.log 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32508.log: /var/log/ceph/ceph-client.admin.33690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32508.log.gz 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21418.log 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.33690.log.gz 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32313.log.gz 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33449.log 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32283.log 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.tmp-client.admin.21418.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21418.log.gz 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33449.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.33449.log.gz 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36762.log 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37048.log 2024-09-16T04:04:00.722 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32283.log.gz 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32179.log 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36762.log.gz 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37048.log.gz 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34399.log 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37569.log 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32179.log.gz 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34399.log.gz 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35596.log 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30953.log 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37569.log.gz 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37987.log 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35596.log.gz 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30953.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30953.log.gz 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33384.log 2024-09-16T04:04:00.723 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36970.log 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37987.log.gz 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37991.log 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.33384.log.gz 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36970.log.gz 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32770.log 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33797.log 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37991.log.gz 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36250.log 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32770.log.gz 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30948.log 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33797.log.gz 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36250.log.gz 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37602.log 2024-09-16T04:04:00.724 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37517.log 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30948.log.gz 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35650.log 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37602.log.gz 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37517.log.gz 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26384.log 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31765.log 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35650.log.gz 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26384.log.gz 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36132.log 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31833.log 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31765.log.gz 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34050.log 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36132.log.gz 2024-09-16T04:04:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31833.log.gz 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31865.log 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37075.log 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34050.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.34050.log.gz 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38843.log 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31865.log.gz 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38583.log 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37075.log.gz 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38843.log.gz 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34689.log 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36030.log 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38583.log.gz 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34689.log.gz 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37979.log 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38505.log 2024-09-16T04:04:00.726 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36030.log.gz 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37979.log.gz 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33114.log 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38505.log.gz 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38869.log 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33114.log.gz 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38869.log.gz 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37963.log 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36102.log 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32205.log 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37963.log.gz 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36102.log.gz 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32905.log 2024-09-16T04:04:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36059.log 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32205.log.gz 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32905.log.gz 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31850.log 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30928.log 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36059.log.gz 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31850.log.gz 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31856.log 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36684.log 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30928.log.gz 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31856.log.gz 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34840.log 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32611.log 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36684.log: 26.3%/var/log/ceph/ceph-client.admin.34840.log: -- replaced with /var/log/ceph/ceph-client.admin.36684.log.gz 2024-09-16T04:04:00.728 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34840.log.gz 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34583.log 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37985.log 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32611.log.gz 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34583.log.gz 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37127.log 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37205.log 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37985.log.gz 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37127.log.gz 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34873.log 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34134.log 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37205.log.gz 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35301.log 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.34873.log.gz 2024-09-16T04:04:00.729 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30959.log 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34134.log.gz 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35301.log.gz 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36375.log 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38453.log 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30959.log.gz 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36840.log 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36375.log.gz 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33676.log 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38453.log.gz 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37231.log 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36840.log.gz 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38817.log 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33676.log.gz 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35140.log 2024-09-16T04:04:00.730 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37231.log.gz 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38817.log.gz 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35229.log 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35009.log 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35140.log.gz 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35229.log.gz 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32153.log 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38375.log 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35009.log.gz 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32153.log.gz 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38531.log 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30757.log 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38375.log.gz 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38531.log.gzgzip 2024-09-16T04:04:00.731 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37596.log 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30757.log.gz 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30827.log 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38791.log 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37596.log.gz 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35305.log 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30827.log.gz 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38791.log.gz 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33543.log 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39051.log 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35305.log: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.35305.log.gz 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33543.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.33543.log.gz --verbose 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr: -- /var/log/ceph/ceph-client.admin.37257.log 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39051.log.gz 2024-09-16T04:04:00.732 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30952.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35467.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37257.log.gz 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30952.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30952.log.gz 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35467.log.gz 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31354.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37309.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.31354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31858.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31354.log.gz 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37309.log.gz 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32744.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5/var/log/ceph/ceph-client.admin.31858.log: --verbose -- /var/log/ceph/ceph-client.admin.30963.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31858.log.gz 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33275.log 2024-09-16T04:04:00.733 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32744.log.gz 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30963.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31821.log 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30963.log.gz 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39261.log 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33275.log.gz 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37179.log 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31821.log.gz 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38947.log 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39261.log.gz 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37562.log 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37179.log.gz 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31844.log 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38947.log.gz 2024-09-16T04:04:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37562.log.gz 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32504.log 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35410.log 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31844.log.gz 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32648.log 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32504.log.gz 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36491.log 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.35410.log.gz 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36176.log 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32648.log.gz 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37439.log 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36491.log.gz 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36176.log.gz 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31363.log 2024-09-16T04:04:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37593.log 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37439.log.gz 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38020.log 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31363.log.gz 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37593.log.gz 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30879.log 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30961.log 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38020.log.gz 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38479.log 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30879.log.gz 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30961.log.gz 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30853.log 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38479.log.gz 2024-09-16T04:04:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38713.log 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30940.log 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30853.log.gz 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36128.log 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38713.log.gz 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38003.log 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30940.log.gz 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36128.log.gz 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31751.log 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38003.log.gz 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30807.log 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31751.log.gz 2024-09-16T04:04:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26265.log 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30964.log 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30807.log.gz 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26265.log.gz 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36736.log 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33973.log 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30964.log.gz 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36736.log.gz 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34335.log 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33973.log.gz 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37361.log 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34335.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34335.log.gz -5 2024-09-16T04:04:00.738 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39365.log 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32853.log 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37361.log.gz 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36892.log 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39365.log.gz 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32853.log.gz 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38635.log 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36892.log.gz 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38973.log 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38635.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38635.log.gz 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33139.log 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38973.log.gz 2024-09-16T04:04:00.739 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31879.log 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.33139.log.gz 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31376.log 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31879.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36206.log 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31879.log.gz 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31364.log 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31376.log.gz 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38661.log 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36206.log.gz 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31364.log.gz 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37101.log 2024-09-16T04:04:00.740 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38401.log 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38661.log.gz 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30965.log 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37101.log.gz 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38002.log 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38401.log.gz 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30965.log.gz 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31775.log 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36380.log 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38002.log.gz 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33058.log 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31775.log.gz 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32795.log 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36380.log.gz 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34508.log 2024-09-16T04:04:00.741 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33058.log.gz 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32795.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr: 25.0% -- replaced with /var/log/ceph/ceph-client.admin.32795.log.gz 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34508.log.gz 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36550.log 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38001.log 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37573.log 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36550.log.gz 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37977.log 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38001.log.gz 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35816.log 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37573.log.gz 2024-09-16T04:04:00.742 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30950.log 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37977.log.gz 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35085.log 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35816.log.gz 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34025.log 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30950.log.gz 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36814.log 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.35085.log.gz 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32361.log 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34025.log.gz 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35061.log 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36814.log.gz 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32361.log.gz 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37592.log 2024-09-16T04:04:00.743 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35061.log.gz 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37598.log 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36866.log 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37592.log: /var/log/ceph/ceph-client.admin.37598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37592.log.gz 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31776.log 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37598.log.gz 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36866.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.36866.log.gz --verbose 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr: -- /var/log/ceph/ceph-client.admin.31324.log 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33593.log 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31776.log.gz 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36299.log 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31324.log.gz 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36323.log 2024-09-16T04:04:00.744 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33593.log.gz 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36601.log 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36299.log.gz 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34339.log 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36323.log.gz 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36601.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36601.log.gz 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35707.log 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34339.log.gz 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31255.log 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35707.log.gz 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33456.log 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34664.log 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31255.log.gz 2024-09-16T04:04:00.745 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33997.log 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33456.log.gz 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34664.log.gz 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31379.log 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37283.log 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33997.log: /var/log/ceph/ceph-client.admin.31379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38921.log 0.0% 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33997.log.gz 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31379.log.gz 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37283.log.gz 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39287.log 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34277.log 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38921.log.gz 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33846.log 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39287.log.gz 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34478.log 2024-09-16T04:04:00.746 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34277.log: /var/log/ceph/ceph-client.admin.33846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38009.log 2024-09-16T04:04:00.747 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33846.log.gz 2024-09-16T04:04:00.747 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34277.log.gz 2024-09-16T04:04:00.747 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34478.log.gz 2024-09-16T04:04:00.747 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37153.log 2024-09-16T04:04:00.747 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36710.log 2024-09-16T04:04:00.748 INFO:teuthology.orchestra.run.smithi188.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T04:04:00.753 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38009.log: /var/log/ceph/ceph-client.admin.37153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32580.log 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38009.log.gz 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37153.log.gz 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32231.log 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36710.log.gz 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32580.log.gz 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35170.log 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33618.log 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32231.log: /var/log/ceph/ceph-client.admin.35170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30701.log 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32231.log.gz 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.35170.log.gz/var/log/ceph/ceph-client.admin.33618.log: 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr: 24.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31365.log 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33618.log.gz 2024-09-16T04:04:00.754 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30701.log.gz 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32352.log 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31332.log 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31365.log: /var/log/ceph/ceph-client.admin.32352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35972.log 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31365.log.gz 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32352.log.gz 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31332.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31332.log.gz 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33388.log 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32606.log 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35972.log: /var/log/ceph/ceph-client.admin.33388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36944.log 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35972.log.gz 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33388.log.gz 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31377.log 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.32606.log.gz 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30782.log 2024-09-16T04:04:00.755 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36944.log: /var/log/ceph/ceph-client.admin.31377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34554.log 2024-09-16T04:04:00.756 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36944.log.gz 2024-09-16T04:04:00.756 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31377.log.gz 2024-09-16T04:04:00.756 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30782.log.gz 2024-09-16T04:04:00.756 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34554.log: 25.2% -- replaced with /var/log/ceph/ceph-client.admin.34554.log.gz 2024-09-16T04:04:00.758 INFO:teuthology.orchestra.run.smithi039.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T04:04:00.803 INFO:teuthology.orchestra.run.smithi117.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T04:04:00.885 INFO:teuthology.orchestra.run.smithi039.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-16T04:04:00.919 INFO:teuthology.orchestra.run.smithi037.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T04:04:01.002 INFO:teuthology.orchestra.run.smithi039.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T04:04:01.007 INFO:teuthology.orchestra.run.smithi037.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-16T04:04:01.020 INFO:teuthology.orchestra.run.smithi188.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-16T04:04:01.187 INFO:teuthology.orchestra.run.smithi037.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-16T04:04:01.216 INFO:teuthology.orchestra.run.smithi188.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-16T04:04:01.399 INFO:teuthology.orchestra.run.smithi188.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T04:04:01.423 INFO:teuthology.orchestra.run.smithi117.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-16T04:04:01.432 INFO:teuthology.orchestra.run.smithi188.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T04:04:01.433 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-09-16T04:04:01.433 INFO:teuthology.orchestra.run.smithi188.stderr:real 0m0.792s 2024-09-16T04:04:01.433 INFO:teuthology.orchestra.run.smithi188.stderr:user 0m2.459s 2024-09-16T04:04:01.433 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m0.084s 2024-09-16T04:04:01.465 INFO:teuthology.orchestra.run.smithi117.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T04:04:01.509 INFO:teuthology.orchestra.run.smithi117.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T04:04:01.519 INFO:teuthology.orchestra.run.smithi039.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T04:04:01.547 INFO:teuthology.orchestra.run.smithi117.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-16T04:04:01.548 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-16T04:04:01.548 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m0.919s 2024-09-16T04:04:01.548 INFO:teuthology.orchestra.run.smithi117.stderr:user 0m3.420s 2024-09-16T04:04:01.548 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m0.124s 2024-09-16T04:04:01.564 INFO:teuthology.orchestra.run.smithi037.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T04:04:01.632 INFO:teuthology.orchestra.run.smithi039.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-16T04:04:01.633 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-16T04:04:01.633 INFO:teuthology.orchestra.run.smithi039.stderr:real 0m1.028s 2024-09-16T04:04:01.633 INFO:teuthology.orchestra.run.smithi039.stderr:user 0m2.617s 2024-09-16T04:04:01.633 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m0.099s 2024-09-16T04:04:01.796 INFO:teuthology.orchestra.run.smithi037.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T04:04:01.797 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T04:04:01.798 INFO:teuthology.orchestra.run.smithi037.stderr:real 0m1.204s 2024-09-16T04:04:01.798 INFO:teuthology.orchestra.run.smithi037.stderr:user 0m3.317s 2024-09-16T04:04:01.798 INFO:teuthology.orchestra.run.smithi037.stderr:sys 0m0.164s 2024-09-16T04:04:01.798 INFO:tasks.ceph:Archiving logs... 2024-09-16T04:04:01.798 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi037/log 2024-09-16T04:04:01.799 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:04:02.443 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi039/log 2024-09-16T04:04:02.444 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:04:02.782 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi117/log 2024-09-16T04:04:02.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:04:03.207 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi188/log 2024-09-16T04:04:03.207 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:04:03.570 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T04:04:03.579 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T04:04:03.579 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:04:03.582 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:04:03.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:04:03.586 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:04:03.720 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T04:04:03.728 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T04:04:03.728 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:04:03.731 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:04:03.733 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:04:03.736 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:04:03.742 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:04:03.742 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-09-16T04:04:03.742 INFO:teuthology.orchestra.run.smithi037.stdout:+hv01.front.sepi 67.205.162.81 3 u 5 64 177 0.072 +0.340 0.650 2024-09-16T04:04:03.742 INFO:teuthology.orchestra.run.smithi037.stdout:+hv02.front.sepi 63.231.80.2 3 u 2 64 177 0.096 -1.434 0.487 2024-09-16T04:04:03.742 INFO:teuthology.orchestra.run.smithi037.stdout:*hv03.front.sepi 74.6.168.72 3 u 10 64 177 0.095 -0.822 0.949 2024-09-16T04:04:03.742 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:04:03.743 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:04:03.743 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-16T04:04:03.743 INFO:teuthology.orchestra.run.smithi039.stdout:+hv01.front.sepi 67.205.162.81 3 u 65 64 77 0.075 +0.195 1.274 2024-09-16T04:04:03.743 INFO:teuthology.orchestra.run.smithi039.stdout:+hv02.front.sepi 63.231.80.2 3 u 3 64 177 0.077 -1.480 1.084 2024-09-16T04:04:03.743 INFO:teuthology.orchestra.run.smithi039.stdout:*hv03.front.sepi 74.6.168.72 3 u 9 64 177 0.108 -0.103 1.044 2024-09-16T04:04:03.743 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:04:03.745 INFO:teuthology.orchestra.run.smithi117.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:04:03.746 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================== 2024-09-16T04:04:03.746 INFO:teuthology.orchestra.run.smithi117.stdout:*hv01.front.sepi 67.205.162.81 3 u 3 64 177 0.133 -0.168 0.880 2024-09-16T04:04:03.746 INFO:teuthology.orchestra.run.smithi117.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 64 177 0.072 -1.713 0.932 2024-09-16T04:04:03.746 INFO:teuthology.orchestra.run.smithi117.stdout:+hv03.front.sepi 74.6.168.72 3 u 67 64 77 0.129 -0.085 0.984 2024-09-16T04:04:03.746 INFO:teuthology.orchestra.run.smithi117.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:04:03.748 INFO:teuthology.orchestra.run.smithi188.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:04:03.748 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================== 2024-09-16T04:04:03.748 INFO:teuthology.orchestra.run.smithi188.stdout:*hv01.front.sepi 67.205.162.81 3 u 5 64 177 0.168 +0.004 1.026 2024-09-16T04:04:03.749 INFO:teuthology.orchestra.run.smithi188.stdout:+hv02.front.sepi 63.231.80.2 3 u 3 64 177 0.080 -3.312 1.446 2024-09-16T04:04:03.749 INFO:teuthology.orchestra.run.smithi188.stdout:+hv03.front.sepi 74.6.168.72 3 u 1 64 177 0.106 -0.662 0.732 2024-09-16T04:04:03.749 INFO:teuthology.orchestra.run.smithi188.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:04:03.750 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T04:04:03.758 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T04:04:03.759 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T04:04:03.767 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T04:04:03.776 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T04:04:03.784 INFO:teuthology.task.internal:Duration was 869.500227 seconds 2024-09-16T04:04:03.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T04:04:03.795 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T04:04:03.796 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:04:03.798 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:04:03.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:04:03.803 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:04:03.840 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T04:04:03.841 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T04:04:03.841 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:04:03.886 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-09-16T04:04:03.886 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:04:03.896 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-09-16T04:04:03.897 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:04:03.907 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-09-16T04:04:03.907 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:04:03.918 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T04:04:03.918 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:04:03.930 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:04:03.942 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:04:03.949 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:04:03.984 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T04:04:03.985 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:04:04.002 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:04:04.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:04:04.017 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:04:04.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T04:04:04.094 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T04:04:04.094 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:04:04.105 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:04:04.116 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:04:04.129 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:04:04.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T04:04:04.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:04:04.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:04:04.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:04:04.162 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-09-16T04:04:04.167 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-09-16T04:04:04.173 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:04:04.181 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-09-16T04:04:04.197 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-09-16T04:04:04.257 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:04:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:04:04.263 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:04:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:04:04.267 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:04:04.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:04:04.272 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:04:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:04:04.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T04:04:04.313 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T04:04:04.314 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi037 2024-09-16T04:04:04.314 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:04:04.336 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi039 2024-09-16T04:04:04.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:04:04.359 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi117 2024-09-16T04:04:04.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:04:04.380 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905926/remote/smithi188 2024-09-16T04:04:04.381 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:04:04.403 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T04:04:04.404 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:04:04.406 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:04:04.409 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:04:04.421 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:04:04.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T04:04:04.459 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T04:04:04.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T04:04:04.561 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T04:04:04.561 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:04:04.564 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:04:04.566 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:04:04.568 INFO:teuthology.orchestra.run.smithi037.stdout: 658642 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:04 /home/ubuntu/cephtest 2024-09-16T04:04:04.568 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:04:04.570 INFO:teuthology.orchestra.run.smithi039.stdout: 658632 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:04 /home/ubuntu/cephtest 2024-09-16T04:04:04.572 INFO:teuthology.orchestra.run.smithi117.stdout: 658642 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:04 /home/ubuntu/cephtest 2024-09-16T04:04:04.574 INFO:teuthology.orchestra.run.smithi188.stdout: 658729 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:04 /home/ubuntu/cephtest 2024-09-16T04:04:04.576 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T04:04:04.586 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T04:04:04.608 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 869.5002269744873 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T04:04:04.608 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:04:04.692 INFO:teuthology.run:pass